1.18.2013

By Request: Kirk's woeful and unruly back-up strategy.

If you really love an image then make a really nice print and store it well.....




I once sat through a lecture by Vincent Laforet which included a half an hour sidebar on his back up strategy. At the time he was using an Apple iServe system which is basically a huge box filled with expensive and redundant hard drives. Everything he filmed or photographed would be backed up on three different RAID arrays and then spooled to Tandberg tape back-up systems. Two copies would be made of the tapes and stored in different locations. Laforet postulated that, at the time, he had several hundred terabytes of information stored and that his monthly cost for cooling the storage infrastructure (just cooling!!!) was about $800. He went further and told us that he would "migrate" the data to newer HDs on a fixed schedule. A "back of the envelope" calculation quickly told the audience that this methodology would require, in short order, a full time employee just to migrate and service the stored information. And can you imagine the number of man hours expended in cataloging each image and video sequence?  I squirmed in my seat. I'd seen the work and like every other artist I know the keepers probably amounted to 1% (generous) of the total stored. That's a lot of seconds to hang on to for posterity...

Of course, when I got back to my humble office in west Austin I felt like an inadequate IT provisioner. Until I remembered that I'm a freakin' artist and not a server maintenance worker....

All my storage philosophy derives from the formative years I spent (about 20) shooting film. Here's how we did back up in the film days:  Shoot film > edit film and throw away the rejects, in the garbage, gone. > Place film in archival sheets and label with description and job number. > After film was used by client and returned, place in filing cabinet by job number. > Try not to set fire to anything in the studio wherein flames might convey to said filing cabinets. > Always make a point to live far above the 100 year flood zone.  While the stakes and our egos were no where as big then as they seem to be now we did routinely make a few "dupes" (film duplicates) of our best work. Not necessarily the profitable work but the work we liked best. We'd use repro dupes if we were selling images as stock. Better to lose an expensive dupe than a irretrievable original. But I'm thinking that maybe we did 100 dupes a year. Max.

There was no separate, off site storage because the bulk of the material existed only as single original pieces of film.

Now we fast forward to now. Seems like everyone I know just puts their images on a hard disk and then makes a second copy on another hard disk and calls it a day. I can't imagine a scarier way to do the business. It's probably a prejudice from the days when our images existed in the physical realm but I need to have one component of my back up that I can hold in my hands.

So, here's how I do it:

I come back from a shoot and I open Lightroom. I set up the import menu to copy the files to an external hard drive. I click the little box to make a second back up to a second external hard drive. I name the folders on both hard drives the same. Before I hit "import" I go through the image thumbnails on the screen and edit out as many as I possibly can. The more images I dump upfront the fewer time and space resources I use up on the back end. I don't need to store everything I ever shot.

Before I hit "import" I make sure I fill out the metadata fields so I can automate most of the organization of the images, after the fact.

I hit "import" and my computing machine sucks the raw images from my memory cards into these waiting folders. When the process is complete I take the memory cards (all SDs, in my case) and put them into a job envelope. The un-erased cards become my third back up, temporarily. If lightning strikes the studio and demolishes the hard drives through some electronic voodoo I'll still have the original files on the cards to reconstruct from.

I buy my external hard drives in pairs. Currently, on the glass topped desk there are two Western Digital 3 Terabyte drives plus a 2 terabyte Time Machine drive that backs up system software and applications.

Next step: I edit in Lightroom. Eventually I'll output the finals from Lightroom into a folder of images that goes to the client. The images are profiled for specific uses, as requested by my clients. Sometimes I am the client.

When I've done everything I'm going to do for a job I take the original edited raws and all new Jpeg or Tiff variations and put them into one folder with the job name. I might even include a "read me" file if the job is complex and multi-faceted. Then I burn two sets of DVD's of all those files. On a large job the folder often exceeds the 4.4 gigabyte limit of the DVD's so I use an app called "Folder Splitter" to bust the big folder into manageable, smaller (4.4gig) folders. A big job, like the recent Dell World 2012 job, might have four or five folders. Doesn't matter, we burn them till we've got them. Two copies of every folder. If the job has a reasonable expectation of a long life or the images are my own art for my own pleasure, or family memories, one copy gets burned onto a Verbatim Gold DVD with a (advertising) promise of a 100 year life.

I've been using DVDs for seven years now and have not had a single failure. Surely, I have just jinxed myself...

We make very clear, in our conversations, contracts and delivery paperwork, to clients that we are not responsible for storage of the images beyond three years. If images are mission critical to them we advise that they make back up copies to store in their servers. Just like printers we are not selling them plates and print blankets for eternity we are selling one time use as specified by contract. Once images are delivered and applied the contract has been fulfilled. While we make every attempt to keep archives of everyone's files for up to seven years we are not in the "storage" business. And we don't want to be in the storage business.

From time to time we go through the filing cabinets and throw stuff out. The economic landscape seems especially volatile right now. Does it really make sense to hang on to those headshots we did for Dr. Koop.com? Or any one of a couple dozen tech start ups that are long gone? I don't think so.  I just dumped about six linear inches of DVDs that were headshots of executives from around 2002-2004. Those are images that were never going to be ordered again.

I'm not doing any real storage on the web. Well, I have over 100,000 images up on the Smugmug servers but those images are also resident on my disks and DVDs here as well. And none of the ones on Smugmug are higher res than 1800 pixels wide.

When storage takes more time than coffee or shooting or lunches you've just switched over to a different sort of undertaking than photography. I'm sure many of you have more advanced methods and I'm sure I'm falling short of some people's perceptions of "industry standards" but the question was, "What do I do for back-up?"  Not "what is the best practice in the universe for archival backup over the millenia?"  Sound reasonable?

Thanks, Kirk

24 comments:

robert quiet photographer said...

Interesting, thanks. And yes, always print! One each ten, twenty or fifty, smaller or larger I always suggest my friends to print the best files. Put them in an album or in a shoebox which they will open with joy (and surprise) in a few years...
robert

Doug said...

Interesting. I'm using Zenfolio for high res backup (edited jpegs from every job, full res), as my offsite backup and archive. More peace of mind that way. I have a separate private folder of my must have "would put me back to square one without" files, of about 200 of my favorites. In house everything is on duplicate hd's.

Dogman said...

I woke up one morning and discovered my Time Machine external hard drive was not working properly. In addition to that, my second external hard drive had completely lost the partition that contained my backed up photos. Super bummer. I still don't know what happened but I no longer trust Western Digital external hard drives--or any external hard drive for that matter. And my faith in technology in general is pretty shaky these days. My 40-year old negatives and B&W prints still look good today with minimal attention to storage. The slides, with the exception of the Kodachromes...not so well.

Since I'm not a pro, I shoot conservatively, ruthlessly edit my shots in camera and later after importing into the computer and I've only been shooting digitally for less than 5 years, I don't have a ton of digital photos that I really consider worth preserving. I bought three portable hard drives and I'm now using them for back up. Furthermore, I put in a good supply of cotton rag fine art photo paper and Epson ink and started archiving my favorite photos in good old fashioned hard copies. While there is no way to insure against the loss of everything no matter what precautions are taken, I'm a believer in multiple back ups these days.

Lanthus Clark said...

I keep it simple. If I really really like it... I print it. Sometimes big.

When the aliens invade or some fool finally presses the red button and ends the life of all electronic life on Earth as we know it (then hopefully) the prints will remain.

Meanwhile I have several hard-drives full to the brim. Damn.

Mark said...

What do you think about _not_ backing up the Raw files and just using a lossless format like tiff? In addition to data loss, I am concerned about format incompatibility in the future. For my non-professional uses, I almost never go back and retouch images, and even if I did, I'd probably be happy with a baked tiff in most cases.

AdamR said...

I'm not very good about throwing out the bad shots before backing up. I delete the really obvious bad ones (missed focus, motion blur, etc) right off the camera but usually I just copy what's left on the card to the hard drive and don't do much deleting once it gets there.

Anyway, here's my procedure as an amateur who just shoots for fun and doesn't shoot a large number of shots. Copy photos to hard drive. Import in Lightroom, add metadata, flag the ones I like and edit those. If there's one I really like I mark it for printing later. Before I format the memory card I connect an external hard drive and backup. External hard drive goes into my fire proof safe. I'm sure the hard drive would be damaged in a fire but the flames wouldn't get to it :) Every couple of months (when I get an Mpix coupon) my wife and I get 8x10's of our favorite photos. These either get framed or go in the safe. That way if both hard drives die we at least have a physical copies of our best/favorite work. I also upload quite a few pictures to Flickr to share with friends and family but I don't consider that a backup.

My biggest problem is my folder structure. Right now I make a folder for the year with subfolders for each month. In those folders are folders for each time I go out shooting. That's fine if I remember I took a picture at the trail building event in April of 2009 but less so for "that cute picture of my dog from the park a couple years ago."

Adam

Kirk, Photographer/Writer said...

To be honest, jpegs at the lowest compression, are pretty darn good for most uses. I have raw files going back to 1999 that are still readable. As I understand it raws are really Tiffs with some extra voodoo.

Unknown said...

Interesting to see that you will go to seemingly valueless files from 10 years ago and just throw them out? Did you do that with your film as well, or do you see the digital files as not being as valuable for some reason? Do you see any potential value in maintaining the archive for historical preservation purposes? For corporate clients that you consistently work for throughout the years, do you see any potential monetary value down the road in collecting, packaging and offering up this historical record of their business or institution?

I've been thinking a lot lately about the historical archive value of the work that professional photographers build over the years. I'd be interested to hear your take on that??

John Gillooly
Boston

Kirk, Photographer/Writer said...

I routinely go through my film and digital files and throw away older, strictly commercial images that I think have no residual value. In the old days great photography was a rarity and historical preservation was a whole different story. Now everything is well documented and widely available. If I perceive an image of mine to have value I keep it. Personal work I keep. But I learned in the ad business that stuff that's really cool today is worthless as soon as the product is off the market. Let me ask you this: I have a bunch of technically correct but uninspiring images of the Eiffel Tower from a series of trips to Paris. If I decide they are lame and need to be expunged will the world morn the loss of my "historical" archive? I think the world will be well enough served by the hundreds of millions of other Eiffel Tower images created every year (month). I had images of employees from a computer company that's gone bankrupt. I can't sell the images because they are not blanket model released (and there is no market for them). The company no longer exists. Some of the people in the files are quite dead. Do I have some sort of responsibility to posterity to preserver hundreds of thousands of strictly commercial images because in some alternate universe they may have value after I am long dead?

Here's the deal that no pro or artist ever wants to admit: They may not be famous enough or travel in the right circles to the point that their work will be collected and curated after their deaths. Did you know that of the millions of children that play sports in high school 96% of parent BELIEVE that their child will get some sort of athletic scholarship from a college? Did you know the actual figure is .01%?

I think it's very selfish not to prune your work as you go. What you do by making excuses for you ego is to leave your heirs with a guilty mess. Boxes and boxes of slides and print film and CDs and DVDs and drawers full of hard drives that they feel duty bound to do something with in spite of the irrelevance of the work. Imagine your kid spending his summers trying to track down who the fat guy on page 32 of your Zecor Technologies contact sheet file is and what his relevance to American History might be in a misguided attempt to honor their parent's legacy. Criminal.

Unless you are at the very top of your niche my personal belief is that only a handful of images you make will have any relevance to general culture after your death. When it comes to images magazines, newspaper and civic archive are already trying to fill Dixie Cups from Tsunamis. It's not like the old days when only Henry the small town photographer had any chronicles of the towns history. And with digital the value drops by the second after the images squirt from the cameras. Pictures of your kid? Priceless to you. Worthless to the world at large. Even more so if the work didn't profoundly move you in the first place.

John Krumm said...

I thought a Tiff was more like an uncompressed Jpeg. That is, it has whatever contrast, saturation, and sharpness you want baked in, whereas the raw is the raw data (ready to be made into a Tiff or Jpeg).

Anonymous said...

An image of the Eiffel Tower... worthless.
An image of my 6 year old grand daughter with the Eiffel Tower as background...
Priceless... to me. MG Van Drunen

Kirk, Photographer/Writer said...

Exactly.

Anonymous said...

Whatever your backup strategy, be sure to periodically test the backup data. I've see a lot of data lost that people thought was backed up.

Kirk, Photographer/Writer said...

agree. Double check and migrate.

Anonymous said...

I suppose I'm a .01 percenter...full-ride tennis scholarship.

Backing up is a different matter. It has confused and frustrated me for years. That's what you get with an athletic scholarship!

Alex Monro said...

Verbatim Gold DVDs may well keep data for 100 years. But whether there'll be DVD drives around to read them in 100 years - doubtful. Anyone remember 5.25" floppy disks, the mainstay of data backup for most personal computer users 30 odd years ago?

A long term backup strategy is going to involve copying all the data to the current latest and greatest media format every 20 odd years or so,

Murray Davidson said...

A priceless article on backup that's been around for years. Enjoy. And take to heart, grasshoppers. The tao of backup at http://www.taobackup.com

Kirk, Photographer/Writer said...

That's a fun read.

Philip Storry said...

I take great pains with backups. Data goes off to a cloud storage provider that I have 300Gb with, but it also goes to two disks on site. Partly because restoring from the disks is faster, and partly because it's just more locations for the data to be on.

But what strikes me most is your comment:
" I squirmed in my seat. I'd seen the work and like every other artist I know the keepers probably amounted to 1% (generous) of the total stored. That's a lot of seconds to hang on to for posterity..."

Yup. I'll be hitting that 300Gb limit soon, and when I do, I've got a choice. Pay more for more storage, or start weeding.

I know exactly which one I'm going to do. And it's not going to hit the wallet... ;-)

Jeff said...

I read this article about a year ago and trimmed 580GB to 76GB in a few weeks.
http://digital-photography-school.com/delete-ruthlessly
I also put photos of family in a separate folder from other pictures.

Unknown said...

Yesterday an earlier file storage system totally failed.
i went to some old CD's and a few refused to open..
i hate this pie in sky about digital..
i make prints regularly.
i am sure my old negatives and prints will be around, when Windows Ultra whatever or Apple Worm are all history.
Thank goodness i am no longer a pro.The fear of lost images, corrupted files not in my picture. Yes i do use digital, but make prints.
Have 3 back up drives for the "pie in sky" images...

Kirk, Photographer/Writer said...

Jason, I think we have to change the business models to reflect the difference in digital and film re: archival keeping. We have to either add fees for periods of storage or limit our liability to a one time deliverable to clients....

John F. Opie said...

I swap out my hard drives every two years, using the so-called 24/7 drives (high-end mechanics, consumer-grade electronics) from WD, Seagate or Samsung. Always buy two identical drives, using the previous set of drives in external boxes as backups (copy and remove). I have one internal drive for photographs (2TB) and an external drive dedicated to mirroring that one. I don't trust cloud solutions.

Then again, I have little commercial work (sold a few Gigapans recently), but wouldn't sleep well without backups. Film photography of more than 40 years (35mm, 120 and 4x5) waits to be scanned in when I have the time...

Ruby said...

“If you really love an image then make a really nice print and store it well...” – Yes, yes. A complete and reliable backup strategy shouldn't lack physical copies. If one can follow the 3-2-1 approach: three copies, two media and a physical separation between the copies, much better. Ruby@Williams Data Management