Image Data Storage and Backup

It’s a sad fact of life but here’s the news:

Your computer System disc is ‘doomed to failure’ at some point in time simply because it has to work VERY HARD.

Desktop machines can be fitted with more than 1 internal drive, but laptops and ‘all-in-ones’ like the iMac can usually only use one internal drive.

Non-system, or external drives work A LOT LESS and so make for a more secure home for your precious original camera files.

So when it comes to Image Data Storage and Backup, the first thing I would advise is that these external drives are where our image files MUST live.

Applications such as Lightroom ‘live’ on your system drive – they have to.

With regard to Lightroom your next most valuable commodity is your CATALOGUE because of all the valuable processing & IPTC data it contains.

By default this is stored on your SYSTEM drive – usually in ‘PICTURES or My Pictures’ – and so from an Image Data Storage and Backup perspective it is actually in the worst place possible!

Your Lightroom catalogue comprises 3 files:

1. The actual .lrcat file itself
2. The catalogue previews.lrdata file
3. The catalogue smart previews.lrdata file

Note: Lightroom will NOT function unless all 3 of these files are in the same place!

Personally I never use 3 – smart previews – do I Mr. Stott (private wise-crack)!  But the file still needs to be there for Lightroom to function.

The .lrcat file, considering the information it carries, isn’t really all that large.  But the accompanying previews.lrdata file can get MASSIVE!

You will see all sorts of crap talked about catalogue location on the internet – mainly about the speed benefits of locating it on your fastest drive.  The fastest drive on most machines is commonly the system drive – especially when computers have SSD drives fitted.

But not only is this the most vulnerable location for the main .lrcat file, the system drive is now open to being swamped by the size of the previews.lrdata file – resulting in your system speed decreasing because applications have not got enough free disk space in order to function correctly.

So catalogue location is always a trade-off in some way or other, but if you take my advice keep your catalogue (or catalogues) well away from your system drive.

Image Data Storage and Backup – Simple Image Data Storage and Backup Using Lightroom Import.

In the simple setup above, I have two identical hard drives plugged into the MacBook Pro and I’m using drive A for working storage and drive B as a backup.  My Lightroom catalogue is on drive A.

This is my standard setup for workshops, overseas trips etc.

I use the Import Dialogue in Lightroom to import the images to the A drive, and thus into my catalogue.

I COULD also select the option (see blow) to make simultaneous backups of the image files onto drive B:

However, this means that my A drive contains my catalogue, image previews, raw files, .xmp files and image adjustments, and any tiffs or jpegs I make from the raw files.  But the B drive just contains copies of the raw files and nothing else.

There is nothing wrong with working like this – after all the most important thing is to get those raw files backed up, and this method of working achieves that goal very well.

I could elect to backup the catalogue to the B drive too, but any tiff or jpeg files I made –  though indexed in the backup catalogue – would only exist on the A drive unless I copied them over to the B drive manually.

Image Data Storage and Backup – “A Slightly Better Mouse Trap”?

The way I work is a bit different, and instead of duplicating the raw files to the B drive on import, I CLONE drive A to drive B.

Cloning the A drive to the B drive ensures that ALL the contents of the A drive – and I mean everything – is safely duplicated and backed up to the backup B drive.

Because I work on the Mac system I use Carbon Copy Cloner, and most Windows users I know use Acronis True Image to achieve the same goal.

So, working with this simple system let’s go through the process:

1. Import images from our camera media to the Lightroom catalogue by putting the images on the A drive.

2. Work inside the catalogue – deleting non-keepers, adding meta data, processing etc.

3. When finished working, drive A contains all the files we need/want to keep so the two drives look like this:

As you can see, drive B is empty, and drive A contains all my data.

4. Start our cloning application – in this case Carbon Copy Cloner:

5. Set the source drive (the disk you want to clone) and the destination (where you want the clone to live) in the task pane and start the operation by clicking the ‘clone’ button.

6. While the clone task is running, notice the wording – ‘Comparing files on source and destination’ – this is interesting.  If we add more images and other data to the A drive later and run the task again in order to ‘update the clone’ CCC only adds files that are new together with files that have changed.

This means that new images are added, as are the changes to the Lightroom catalogue and previews, but it does NOT re-write unchanged files.

So the ‘cloning task’ runs much faster when you re-clone!

7. When the clone task is complete you can see that the file structure of the two disks is identical:

CCC creates (unless you tell it otherwise) a ‘Safety Net’ folder.

Say I’m working in the Lightroom catalogue and find a crappy image that I missed on my original ‘cull’.  I can tell Lightroom to delete it from the disk and thus from the catalogue.  But it will still be on the clone image/B drive, and registered within the clone of the catalogue.

So when I run the clone task again the catalogue clone is modified to register the removal of the image, but the actual image file is not deleted from the B drive – instead it gets moved to the ‘Safety Net’ folder (just in case I’ve been an idiot!).

Put simply, I can delete files (or even folders) from the A drive, but they will not be deleted from the B drive but moved to the ‘Safety Net’ folder on the next clone operation.

Good ehh?

Image Data Storage and Backup – Hard Drives and Connections

A vexatious subject for sure, and it all comes down to money!

With the laptop setup at the start of this article, the two hard disks I’m using are 500Gb G-Drives in Thunderbolt carts.

These are fast drives on very fast connections, and I do not notice any speed reduction in catalogue functionality.

I can also swap the drives from Thunderbolt to USB 3.0 carts and plug them directly into the USB 3.0 ports on my Mac Pro (which is basically a modified 2009 machine which won’t accept Thunderbolt bus connections).

For me, USB 3.0 bus speeds are plenty fast enough too,  so plugging that A drive into my Mac Pro and opening the Lightroom catalogue I created in Norway on a MacBookPro gives me pretty much the same performance in my office as I had on location.

I can now import the catalogue and files onto my main office machine, and either keep it as a stand-alone catalogue, or merge it with an existing one:

Image Data Storage and Backup – To Raid or not to Raid?

Desktop computers have space in them for the addition of more hard drives – I have 6 in mine:

1. My system disc, which is a 500Gb SSD
2. 3 TB storage
3. 3 TB storage
4. 1TB partitioned into two 500Gb drives – 500Gb for Photoshop Scratch purposes and 500 Gb bootable system disc backup done with CCC.

Discs 5 & 6 are a RAID 0 pair of 4TB WD Blacks, so they basically behave like an 8TB disc.

So what is RAID, and why use it?

Redundant Array of Independent Disks – RAID – in it’s simplest everyday form comes in two flavours – RAID 0 and RAID 1.

If we take two identical drives we can use our operating system drive management and built-in RAID controllers to configure the two drives to work simultaneously in either a 0 or 1 configuration.

RAID 1, from an Image Data Storage and Backup perspective, is a Godsend because as you write a file to one drive it is mirrored onto the other drive in the RAID pair – in other words INSTANT backup without you having to lift so much as a finger!

RAID 0, from an Image Data Storage and Backup perspective is quite the opposite.  As you write a file to a RAID 0 pair that file is broken into blocks, and half the blocks are written to one drive, the other half to the other drive.

Advantage of RAID 0 – both drives are written at the same time, so the write speed is in effect twice as fast as RAID 1.

Disadvantage of RAID 0 – lose 1 of the two drives and ALL data is lost – ooops!

The speed gain is why I use this 8TB RAID 0 two drive array, but only because I have it  mirrored onto an external 8TB G-Drive 2-bay unit configured in RAID 0 as a permanent backup.

See – I told you money was involved!  Don’t waste your cash on new cameras – spend it all on quality glass and hard drives.  You can never have enough of the latter, and the former is what improves the quality of your images.

And that’s not all – I have two 2TB Seagate expansion drives sitting on top of my desktop machine – both on USB 3.0, one operates as my Mac Time Machine backup drive, and the other is the capture, edit and project storage drive for Camtasia which I use for all my training and YouTube videos.

Now obviously you won’t need anything like the volume of storage I have, and I would certainly not recommend you use RAID 0 pairs of drives either.

But for effective image data storage and backup it is both safer and kinder to your overall system speed to keep both your images and your Lightroom catalogue on a drive or drives that are separate from the drive your applications run on.

Footnote: You’ll come across people on the internet/YouTube who will lead you to believe that they back their images up to ‘the Cloud’.

Consider someone who comes back from Norway after a week on the eagles with ‘yours-truly’ they have been shooting, culling and editing over there and they still might come back with 4000 raw files.  How do you back those up to the cloud?

Let’s now say we cull 90% of those raws, and end up with 400. Let’s now take the best 25% of those and create archival tiff files for print – that’s over 6Gb of files from your average FX camera.  Let’s also take 50% of those 400 shots and make some stock submission full resolution jpegs that may average 5MB each.

So from just that 5 days of shooting we end up with 14GB of raw files, 6.5GB of archive images and over 1GB of jpegs – 21.5GB in all for 5 days shooting.

How long is it going to take to upload 20GB+ over your average internet connection?

If you are a shooter of any kind of volume then Cloud backup is not a viable option in my opinion, because it’s your raw and archive images that are critical, not the jpegs!

Latest YouTube Video:

In case you missed it I uploaded a video to my YouTube channel the other day, processing a low key Barn Owl image in Lightroom and Photoshop and using the Lumenzia plugin to do some very simple but effective localised adjustments.

Overall, the image would be 100% IMPOSSIBLE to create in just Lightroom.

If you found this free content useful and would like to see more plus extras not viewable to the general public then please consider joining my existing patrons over on my Patreon page by clicking the link below.

Many thanks!