Archiving your photos, backing up: a cautionary tale

I made this image to illustrate my 2010 New Year's Resolution to develop an iron-clad backup routine. Here I am almost 3 years later, still bit in the butt by sloppy backup habits. (Hint: don't let this be you too!)  (Copyright 2010 / Andrew Boyd)

I made this image to illustrate my 2010 New Year’s Resolution to develop an iron-clad backup routine. Here I am almost 3 years later, still bit in the butt by sloppy backup habits. (Hint: don’t let this be you too!) (Copyright 2010 / Andrew Boyd)

A Tale of Woe

Backing up your photos–properly archiving everything–is something we all say we’re going to do. But how many of us consistently do it? My complacency on this issue recently turned into the nightmare of all nightmares, and I’m here to tell you about it: kids, don’t try this at home!

I changed out laptops about 4 months ago, trading in my old MacBook Pro for a new one. Along with all of my work-related files for and The Times-Picayune, the old machine contained a monster series of nested folders of my landscape photography, all of which I regularly backed up to an external hard drive using the Mac’s built-in Time Machine application. All good so far, right?

Well, when getting ready to switch to the new machine, I took that big set of landscape photography folders and did a manual additional copy over to an external disk–you can’t be too careful, right? It was around 40 GB of material and I left it to copy overnight. The next day I did some additional cleanup of the machine, getting ready to turn it in.

Fast forward two months, and I needed to print images for the upcoming ‘Fall for Art’ show at the Henry Hood Gallery in Covington, Louisiana. I was looking forward to this process, thinking about some specific images from the Atlantic shore that I wanted to trying printing for the first time. I hooked up my external hard drive and went to the set of nested folders. Clicking the first one open I found….nothing! A blank, empty folder! I quickly tried several more—and found them all to be empty. Somehow, my folder structure had copied overly perfectly, but none of the files! I desperately tried everything I could think of, even pulling out every other external hard drive that I owned, looking for these photographs—none of it was there.

I didn’t know if my overnight copy timed out somehow or if some other gremlin was at work, all I knew after a careful, methodical search of every possibility was that the last year and a half of my landscape work–all of it—was GONE. To make matters infinitely worse, I had replaced my old Time Machine configuration with the new one that I’m now using on the new laptop–effectively throwing away the second, last copy of these images.

A Glimmer of Hope

Despair, depression, desperation: all good words to describe that day. At the end of it all, I held onto one slim, glimmer of a possibility: what if my old machine was still around, and even more unlikely, what if it had not been wiped off and reformatted? I sent an email to our I.T. department and discovered that yes!–my old laptop was sitting on a shelf, untouched since I surrendered it. And yes, out of sympathy for me (and the fact that my friends in I.T. know I’m never been a problem child for them)–they said I could borrow it for awhile.

I knew that the recovery process was fraught with difficulties and not at all a sure bet, but certainly worth the effort. I did a bunch of online research, tried a couple of recovery programs that turned out to be duds before settling on a wonderful, reasonably-priced solution from the fine folks at Wondershare: Wondershare Data Recovery. I’ve used some of their other products and have always been pleased so giving this one a go was not a hard decision.

This is the Wondershare Data Recovery main menu. (Copyright 2013 / Andrew Boyd)

This is the Wondershare Data Recovery main menu. (Copyright 2013 / Andrew Boyd)

To use the Wondershare Data Recovery application, you need the machine you hope to recover files from hooked up to another machine, upon which you have enough free space and will run the program from. Because my old machine was a 4+ year-old Mac and my new machine only a few months old, I opted for a Firewire 800 cable to connect the two. (If you were going to run a recovery between two newer machines, you’d want to do this with a Thunderbolt-to-Thunderbolt data cable since your transfer speeds will be quicker.)

Old computer on the left, new computer on the right: a Firewire-to-Firewire 800 cable connects the two devices together, mounting the old computer as a drive on the Desktop on the new machine, where Wondershare Data Recovery is running. (Copyright 2013 / Andrew Boyd)

Old computer on the left, new computer on the right: a Firewire-to-Firewire 800 cable connects the two devices together, mounting the old computer as a drive on the Desktop on the new machine, where Wondershare Data Recovery is running. (Copyright 2013 / Andrew Boyd)

The process is simple: Have your new machine running and the cable hooked up between the two boxes. Then start the older machine, while holding your finger down on the “T” key. The old machine will boot up as an external disk, viewable on the desktop of your new machine. You’ll see a symbol that I think stands for ‘Firewire’–I think it looks a bit like a symbol for something that’s radioactive–and that will dance slowly around the screen of your old machine while mounted in this way.

Once they are hooked together, launch Wondershare Data Recovery. The main menu will appear with 5 options. I suggest you try the first option–‘Lost File Recovery’–first. That didn’t work for me, so I went to ‘Raw Recovery’ which began the process of creating a new File Allocation Table for all of my deleted material. It finds everything that’s been deleted–Word docs, Pdfs, jpegs, tifs, pngs, various video file formats–and displays a list for you to peruse. In my case, the program located 75,000 jpegs images plus a slew of other material. Once you hit the ‘Recover’ button at the top of the window, get your credit card out because this is when you’ll have to pay for the privilege of using their product.

As of this writing, the price for the fully-functioning Mac version of the program was $89.95 U.S., a bargain by anyone’s measure.

The Nitty Gritty

What I’ve found is that the program is very good at finding most of my original files–the exposures as they were ingested off the compact flash cards to the computer. It’s not so good at locating really large, fully-toned images. I’m not sure what the exact file size limit it, but there definitely is one. Lots of toned images in the 20-25 MB size were located and recovered intact; bigger files, not so much. But at least with the originals to work from, I can go back and retone the images, immensely preferable to having nothing at all! Also, although it found and generated preview icons for layered Photoshop (.psd) files, it was generally unable to open these, so if this is your main recovery need, this program may not work for you.

Recovery in progress. The file formats being found are listed on the left.(Copyright 2013 / Andrew Boyd)

Recovery in progress. The file formats being found are listed on the left.(Copyright 2013 / Andrew Boyd)

So now I’m in the sorting process: all 75,000 jpegs have to be looked at in some fashion. This is actually very interesting: I never realized it before but every time you upload an image or work on it in Photoshop, different sizes get created and saved. You’ll have tiny little thumbnails still there, getting recovered, and other sizes as well. What I ended up doing was taking a batch of the files–the jpegs get saved out in batches of 10,000– doing a sort-by-size for the batch, which the program allows you to do–and throwing away everything smaller than an arbitrary size of 200K. (I figured that something saved as a jpeg of 200K might be useable, anything smaller, probably not.) This would eliminate 6-7000 of the files right away, leaving me with a more manageable 3000 or so to look through.

Here's how the hodgepodge of images shows up after being rescued by Wondershare Data Recovery. You have to go back and recreate a folder structure for your various projects. (Copyright 2013 / Andrew Boyd)

Here’s how the hodgepodge of images shows up after being rescued by Wondershare Data Recovery.
You have to go back and recreate a folder structure for your various projects. (Copyright 2013 / Andrew Boyd)

Even then you’ll find lots of images saved on your computer that you’ve never seen before! Things like small files that are part of a program you might have on your machine and never use, for instance.

I’m doing the sort process in Photo Mechanic, which not only allows me to browse everything, but will let me create and copy files out to new folders on the fly. So I’m slowly recreating all of my main folders, many times combining photo trips by location. When I’m finished I should have most of my old images back, at least in their original, raw state.

So what’s the takeaway? Obviously, that you can’t have too many back ups. Back up early and often is the dumb-as-rocks lesson here for me. I’m experimenting with a few things that I’ll mention as well.

I took about 2GB worth of toned images that I did NOT lose–the print order folder that I had moved over to the new laptop separately–and copied that monster up to my Google Drive home. This took about an hour but did create an additional, safe location for these files. The downside is that Google Drive does not seem to generate preview icons, so you’re left with available files buy now thumbnails.

I also have a Flickr Pro account which allows 1 TB of storage for, right now, a very cheap $25 U.S. per year. (Note: Flickr Pro accounts are in flux as Flickr messes around with their products. Stay tuned.) This does generate thumbnails and will allow big monster copies up to it. I did this with a bunch of material last night and found it worked very well, although this also is a slow, cumbersome process. The main irritant for me with the Flickr account is that the default settings, which I can’t seem to change, start with everything you copy up being publicly viewable and searchable by the world; I want this stuff to be my private archive. This is possible but requires checking off a bunch of boxes on each batch you upload, a pain in the ass, I think.

So my plan at this point is:

1. To use the Mac’s Time Machine utility to do regular backups of my computer;

2. To copy all landscape work out to an additional external hard drive;

3. To upload all finished personal landscape work to my Flickr account;

4. To make DVD copies of the raw files from every shooting trip and store them in a different location.

Have I learned my lesson? I certainly hope so. Only time will tell!

So what’s your photo archiving process? Do you have one? Do you use it regularly? How did you turn it into a habit that was easy to follow? I’d love to hear how others are handling this ever-so-crucial chore. Please let me know!

Hi, I’m Andrew Boyd, a.k.a. The Discerning Photographer, and I hope this post has been interesting and informative. Please leave me a comment about it, let me know what you’d like to see more of on the site! You can also sign up for email delivery of all future articles or my RSS feed. Or subscribe to our Facebook page ,Google+ pageor our Twitter feed. Thanks!–DiscerningPhotog

Posted in: How To

About the Author:

Photographer, videographer and photo editor. Host and creator of The Discerning Photographer web site. Currently a Canon shooter.

9 Comments on "Archiving your photos, backing up: a cautionary tale"

Trackback | Comments RSS Feed

  1. Bayou Bill says:

    Andrew, I think the other moral to your story is that one should never assume that a backup is good. At a minimum, one should periodically do some spot-checking of the backups to make sure the files you think are there really are, and that they’re recoverable. With modern backup programs there are often so many options that it’s easy to fall into the trap of thinking the end result is something that it actually is not.

    Also, I think your #4 procedure of making a DVD copy of the raw files from every shoot is a good, simple, and (usually, but don’t assume it) reliable way of taking care of business and illustrates that the simplest way of doing things is often the best.

    I’m glad you’ve been able to recover your files. I’ve never had to do that for myself but have done it for i.t. clients and friends on numerous occasions, and it’s always horribly time consuming and usually painful in one way or another. Gives real meaning to the old “An ounce of prevention” maxim.

  2. matt rose says:

    Hi Andrew,
    Nice post. So glad you recovered most of your work.
    I don’t injest onto my computer. Instead, I injest onto a desktop hard drive then immediately copy that to a portable hard drive. Then I have the work stored in two places. And I can take the portable copy when I travel just in case some client needs something right away. I also periodically copy important work from my old full hard drives to my current hard drive. I probably should put important personal work on a DVD and after reading your post, I will start doing that.

  3. Hey Andrew,
    glad to hear that most of your images were recovered. I hope that I will not come into that situation, and yes, my backup is not consistent and on a regular basis. I used to have one based on which worked quite nice for 10.6 or so. Unfortunately I stopped after an upgrade and I have not implemented a new one. While I am a fan of CarbonCopyCloner, I intended to set-up a backup using this, but have not done yet. In irregular time intervals I do manual backups using the Finder, time consuming I know.
    I think the easiest solution (in my MacPro) would be at least 2 3TB HD, one to work with, second to backup. Basically I want to leave my RAID system behind and be able to quickly change HDs in case of damage. Additionally I would do a second backup on an external disc via Firewire or eSATA or so. And yes, with Flickr Pro or SmugMug or even your own Server or Cloud Space, a backup images can be made.

    I know, bad, still to be done. Probably simpler and faster than I think.

    Thanks a lot for the reminder and cross my fingers it will not happen to you again!

    Cheers, Bernd

  4. Hey Bernd,
    Great to hear from you! yes, let my tale of woe be your jumpstart to a more robust backup!

  5. Hey Matt,
    Great to hear from you! So you make two copies right away: I like this. You have something that forces you to make at least one copy right away, that sounds like it would solve most of the issues that came up for me.

  6. Well said, Bill! Thanks for your sage comments!

  7. Hey Bill,
    very good point to check the backup. From the technical point of view chasms should be the valid choice here.
    Using DVD as a backup can also be critical. How about a print or slide instead?
    Ok, that’s not really an option considering we want a backup in digital format, but to my knowledge they will live much longer than anything digital at the moment.

    yes, I a a rare reader of your blog and everywhere around. But I still shoot as you can see on flickr. But mostly from our adventures in the mountains.

    You all take care, Bernd

  8. Sean Garrett says:

    I make use of as a cloud back up. I’m on windows not make but I’m sure it works the same way: trickling the upload in the background up to their servers. I liked them since they have a fixed cost per year (important when you’re paying for things in dollars and you’re base currency isn’t) and unlimited storage.

    I also use freefilesync ( )to sync my lightroom catalog betweent he master and a secondary dirve in my PC, master and and external USB and a network drive.

    Just slightly paranoid 😉

  9. I’ve heard good things about carbonite, thanks Sean.

Post a Comment