Data and Storage
An often under-looked at aspect of our digital world
It’s the start of a new year, which for me, means one thing - running through my data storage solution and inspecting its integrity.
Now, you may be wondering, why is a photographer doing sysadmin work?
It’s for a variety of reasons, but the main one being: I produce thousands of RAW images and videos every month. Using a 51mp camera, capturing 4k60fps, audio, and all that means I’m adding on average about 2TB worth of photos and images every year. That’s a LOT of stuff.
Why don’t you delete everything? Surely it’s not necessary to hold onto everything?
I mean, I do delete things - stuff that’s unintentionally out-of-focus, the accidental pictures taken when the shutter gets knocked around, you know all the obvious ones. Except, while many photographers I know go beyond this with a stricter culling process - I like to keep all my images, even the ones that are poorly lit, horribly composed, and missing a subject.
At its very simplest, those photographs are a reminder of the experience; where in the world I was and what I was seeing. At best, I learn something new editing-wise and can come back to it in the future to bring them to the potential I’d envisioned when I made the image. And between the two, it serves as good material for one of my projects, sans.focus, where I showcase all of these “leftover” images that didn’t make the cut in my “professional” portfolio.
So how do I store it?
Well, redundantly and very securely, of course. Data loss is my personal nightmare. Too many times I’ve had an SD card, flash drive, external drive, computer, phone, laptop, or iPad just crash and fail or be physically destroyed for numerous reasons and lost all my hard-earned media. While “redundancy” usually means getting fired by your company, it means a significantly different thing in data security.
Redundancy is what we want, and the more of it we have, the more secure our data is. Here’s my “simple” recipe for data security:
INGREDIENTS -
Camera with dual card slots
2x portable SSD’s
DAS (Direct Attached Storage)
NAS (Network Attached Storage)
Backblaze (Cloud Backup)
Let’s start at the top
I chose a camera with dual card slots so that I can have my media recorded to both cards simultaneously. This creates my first layer of redundancy, as it’s extremely unlikely that both my SD cards fail at the same exact time.
Once I’m back at camp or the room I’m staying in, I’ll immediately make a copy of each card onto its own external SSD.
If I have internet and my computer with me, media will get uploaded to my NAS. If not, this step is skipped. If I have my laptop with me, I’ll start editing using one SSD, keeping the other untouched and in a safe location.
Once I’m back home, I’ll transfer all media to my DAS which then syncs up with my NAS. If I’d already uploaded the media to my NAS in the previous step, the newly copied data on the NAS will be synced with the DAS. The DAS is plugged into my desktop throughout all this, so I can easily transition from working on my laptop to the desktop.
My DAS is also constantly backed up with Backblaze, one of the only subscription services I pay for. While this goes against my ethos of not allowing companies to handle my data, Backblaze does make reasonably believable effort into maintaining user privacy, and they also do so for a very competitive price.
I also don’t delete things from external drives and instead just label them by date and toss them in a secure container once they’ve filled up.
With this method, I’ll always have 4 exact copies of my data. The externals, my DAS, my NAS, and Backblaze. It should be noted that all 4 of these things are stored in entirely different locations. My DAS is at home, my NAS is in my office, Backblaze is in the cloud, and my externals are stored in a secure case at a friend’s house. Barring some sort of cataclysmic global event, there’s extremely little chance that all 4-5 of these solutions fail at the same time and I’m stuck without a method of retrieving my data.
This is my “evolution” of the 3-2-1 backup system, which you should read more about here. It’s a great system and critical place to start for any digital creative, but in time I’ve found that it still leaves a ton of vulnerabilities open which are solved with the addition of the 4th and temporary 5th layer of redundancy via the external drives and SD cards.
It’s important to note that I haven’t mentioned RAID a single time thus far. It’s for a simple reason.
RAID IS NOT A BACKUP.
While they are often used in DAS and NAS, RAID is in every definition of the word, not a backup. It stands for “redundant array of independent drives,” but despite having redundant in the name, a backup MUST be stored in a separate location. Two copies of data stored on the same NAS or DAS then, is you guessed it, not a backup.
So what is RAID? It’s a method of grouping hard drives within a NAS or DAS for a few benefits, the three main ones being speed, protection against drive failure, and short recovery time. There are several different levels of RAID, main ones being 0, 1, 5, 6, 10, and 01. Each one brings a different set of pros and cons, usually a tradeoff between speed, storage capacity, and level of redundancy.
It’s one of those classic pick 2/3 situations.
I won’t bore you further here, so just refer to this website here for an explanation of the different RAID levels
I use RAID 6 on both NAS and DAS due to its emphasis on drive failure tolerance while having decent speed and storage capacity.
And…I think that’s it.
I guess I should say that underneath all of this, I have another home server that backs all of this up for yet another layer of redundancy (and also has its own backup), but that’s another story entirely that we definitely don’t have the time for. To sum it up quickly, it’s basically having my own versions of Netflix, Apple Music, Google Drive, and advanced home automation all in one - without the subscription services, of course. But again, more on that another time, when I talk about how I ditched and left every single subscription service.
Now if you haven’t already lost your mind, because I surely have, here are some resources you can use to start building your own backup system.
Here’s what I specifically use as reference
DAS: OWC 4M2
Drives for DAS: 4x 8TB Western Digital Black M.2 drives
NAS: Synology DS923+
Drives for NAS: 4x Seagate EXOS 12tb
Cloud: Backblaze Personal PC Plan
Portable: Sharge SSD Enclosure, Crucial X9, Samsung T7
Also, the SanDisk Extreme Pro Portable are a very popular solution for external SSD’s - AVOID THEM AT ALL COSTS.
https://nascompares.com/ is a great place to learn about NAS and other NAS related things.
Do note that the creator of the site, while providing extensive in-depth reviews and is a great resource for all things NAS, does take sponsorships and receive free products for review.
Reddit is a goldmine for weird niche stuff like this. The Wiki page on the r/DataHoarder subreddit is an amazing resource for building your own NAS https://www.reddit.com/r/DataHoarder/wiki/index/
https://shucks.top/
A great place to buy hard drives
Though, it should be warned, drives listed on this site require “shucking,” which is buying a regular external drive and ripping the internal drive out of the enclosure. These go on sale more than internal drives do, and usually at larger discounts, so the effort can be worth hundreds of dollars in savings.
https://serverpartdeals.com/
If you’d like to purchase drives normally, this is a fantastic website. They sell both new and refurbished drives.
Don’t shy away from buying used hard drives, they are all certified and work great (I use refurbished drives in ALL of my servers/NAS/HDD based DAS).
When buying hard drives, make sure that they are CMR and not SMR drives. Basically, CMR drives are better. For a more in-depth explanation, read here
Some reputable companies for purchasing NAS or DAS related products:
The Unify Drive UT2 is a VERY interesting product from a new company, but this is their first product and it was released via Kickstarter and relies on a phone app to function, so we’re going to have to see how it holds up in the long term. I still have PTSD recalling my stupid Gnarbox, which was basically this product minus the NAS functionality. Gnarbox has gone bankrupt and their app no longer works, reducing the product to just a glorified paperweight.
I’d like to purchase one, but it’s not a cheap device so I’ll definitely be waiting a few years to see what happens to the company before buying in.
Lastly, a reminder that it’s not so much a matter of IF a drive will fail, but WHEN it will fail. SSD’s are less reliable than regular spinning hard drives in the sense that when they fail, they tend to do so instantly and immediately. With regular spinning drives, there are usually signs such as weird sounds or weird behaviors when accessing files that prompt an inspection, followed by replacement of a drive.
Actually, this sort of brings me onto the point about short recovery times being one of the pros of RAID that I forgot to talk about earlier. With most RAID levels, when one drive in the NAS fails you can simply turn it off, replace the faulty drive, turn it back on and the NAS will automatically copy the files back over and go back to normal. Some NAS will even allow you do this without needing to be shut down.
This ability to quickly repair and maintain your NAS/DAS is one of the largest benefits to RAID. It significantly reduces your downtime when a data loss event occurs, and allows you to get back to work faster. Once you have a ton of storage, it’s also important to consider just how much time it will actually take to copy all those files.
Hint: it’s a VERY long time. I’m currently on day 3 of transferring files while creating a backup of my entire storage system and I’m only about 25% done.
Thank you all for reading, I know this one was a bit dense but I do strongly believe anyone working with digital files should take data storage a little more seriously. Once data is gone, it’s gone, so it’s important to treat it like an important legal document. At least to me.
Hope this helped and hope your year has been going great so far!