Digital Media & Long Term Storage

[Fuzzy 0]

I’m giving a talk at my photo club on archival storage of old photo’s, negatives, and slides, something my company deals with as we sell the materials used for that sort of thing.

Since the majority of people now shoot digital it’s been asked that storage of digital media be included also from a long term point of view. Long term being the info is still accessible to you’re grand kids and their grand kids. This is something I’m not as familiar with although have some papers on it.

I know that not all CD’s/DVD’s/External Hard Drives are created equal – example the gold Kodak Preservation CD/DVD’s are pricey although very high quality and designed specifically for archival use.

What I’d like to hear are
A: Thoughts or experiences good and bad with specific brands or formats of CD's & DVD's (-R -RW, +R etc.)?

B: Any problems with disc’s going bad after burning (couldn’t access the data), if so were they stored properly i.e. not in direct sun or extreme heat/humidity.

C: External Hard Drives, are there any brands that have proven to be more/less reliable than others?

Any info and experiences is greatly appreciated.

Fuzzy
Ambition / Ability: Know the difference.

[PhreeZone 14]

The way I look at it is most digital media is looking for a 10-15 year life before you may need to switch storage media. If you look at the evolution of media in the 60-70's if you wanted to record music for your grandkids to listen to you would have picked vinal or 8 track, both of those now require specialized play back devices that are getting harder and harder to find. If you looked at the 80's and the introduction of tape you would have selected either casettes for music or reel to reels for data storage. Large format tape for data is really hard to find anymore and you'd need a room to store the tapes in. Move to the 90's and the introduction of recordable CD's. This format has turned out to be one of the longer term storage devices so far but its on its way out. at 700 megs a CD you are looking at 350-500 photos filling a CD. If you are the new digital photographer you take a few thousand photos a year at least you are looking at a stack of CD's for your multiple copies you need to keep. DVD storage is better since a DVD will hold 4.7 gig's. That moves most people down to only a dozen or so DVD's each year they need to store. The good news is that DVD players can read CD's so you at least have some backward compatbility here. DVD as a medium only has 5-7 years left in ot since BluRay is replacing it. That means that I'll be upgrading my storage to BluRay in a while. They are already designing the replacement for BluRay. Once thats out BluRay will be dead and they will have something else that I need to move my media to for storage. The cycle never ends.

Portable hard drives all have about the same MTBF (mean time before failure) unless you start looking at some of the special drives that are made for the DOD and other agencies that are paying way more then a consumer would ever pay for a drive.

I burn 4 copies of every archive DVD. 2 go in the fire safe in seperate albums to have backups if one gets scratched, 1 goes in a "working album" that I use and keep at my desk so when I need to see the files again I have them and 1 copy will go in the safety deposit box (once I get one )

Yesterday is history
And tomorrow is a mystery

Parachutemanuals.com

[Fuzzy 0]

Thanks

Keeping multiple copies in multiple locations is something that was brought up at the last meeting (and part of what stemmed the request for more info at the next.

Will definetly bring up equipment obsolesence issue, important to ensure you can access the info on the disc.

Good to know about the portable drives. Guess bad luck just struck as I've had both a Lacie and WD drive fail within a year of purchase - one reason I don't and wouldn't trust them for archiving, but some people do.

Appreciate the info

Fuzzy
Ambition / Ability: Know the difference.

[PharmerPhil 0]

Quote

Thoughts or experiences good and bad with specific brands or formats of CD's & DVD's (-R -RW, +R etc.)?

Although it can be difficult to find out who actually makes a given blank disk, there are methods listed on this web site, and ratings (opinions?) on the quality and viability of various brands of DVD media. IMHO, this is good information for choosing disks for any use other than simply sneaker-netting files around.

http://www.digitalfaq.com/media/dvdmedia.htm

[DSE 3]

buying cheap drives from Costco and similar for archiving is foolish. Buy enterprise-grade drives, this won't be an issue, but having an HDD and optical...you'll likely never run into issues.

[tonyhathaway 0]

I keep my digitals on 2 drives in my mac in ONE place, and 2 western digital external drives at HOME. Everything is on 4 drives. (except the last few jumps I've done as sometimes I get lazy and take a bit to transfer from DZ to home)
Anything else I could do better?

My O.C.D. has me chasing a dream my A.D.D. won't let me catch.

[Fuzzy 0]

Are you saying you have 4 copies of everything, or that it it takes 4 drives to hold everything (sorry if that sounds like a stupid question)?

Four copies in 2 locations sounds pretty safe, just curious though if you are only using external drives as back up or do you also keep copies on CD/DVD.

My main concern with using external drives or even DVD's is that they hold so much if something does go wrong you stand to loose more at one shot.

Another question - somewhat related - How do you catalogue everything so you can find it when needed?

Fuzzy
Ambition / Ability: Know the difference.

[PhreeZone 14]

Cataloging just became a lot easier... use Adobe Lightroom! (or Aperture if you are an Apple person)

Save the files on external meda and back up the catalog files on the same and it lets you catalog and tag every file for later searches. I've built out a tag library of about 75 tags so that I can search by subject, person in it, activity, location, etc. I've only done it to photos I've taken this year but if you tell me that you want a landing photo containing Scott or a RW photo with Shawn in it I can tell you in 5 seconds if I have something that matches those criteria.

Yesterday is history
And tomorrow is a mystery

Parachutemanuals.com

[tonyhathaway 0]

4 copies on 4 drives, 2 locations. nothing on disks. while its true that if ONE drive goes down, it lost. but thats why i have it on others. Actually, not EVERYTHING is on all 4 drives. On one of the drives at home i have basically the best couple of shots from each dive. If i shot 10 photos, and only 1 or 2 are the BEST, I keep those.
As far as logging, I go by date. A photo I shot yesterday is numbered " 091808-01 " The only problem I can't pull up "clouds" "sunset" etc like with what I'm sure lightroom would do. If I got that program I'd have to go through 70 Gb of photos and who knows how many thousands and log them. I just don't have a year to do that!

My O.C.D. has me chasing a dream my A.D.D. won't let me catch.

[PhreeZone 14]

I cataloged 5300 photos (18 gigs) in under an hour and a half thanks to the batch processing in Lightroom. I would select an entire jump worth of photos and tag it with the same core tags (Who, what and where) and then would do one off tagging of the ther files as needed for things like Sunset, exit, etc. Landings took the longest since I could only apply tags to 4-5 photos in a row before needing to select a different tag since I was tagging them on a name basis.

Lightroom 2.0 is availble as a free 30 day demo from Adobe.

Yesterday is history
And tomorrow is a mystery

Parachutemanuals.com

[Laszloimage 0]

I have every single shot of mine existing on two HDDs. Plus every single one of them backed up on DVD as well.
But even you're using HDD it's hard to belive they'll brake at the same time. Unless if there's an huge electormagnetic surge or all the drives are at the same location during a fire. (very high heat also cancels out magnetic data).
Anyway the bottom end is to have your data saved on two different HDDs plus optical (DVD) is pretty dam safe.
-Laszlo-

[jeremy_o 0]

I agree with Phree here. The important thing is to keep up with the times of digital medium. For example, the current hard drives today will physically last, say 4-8 years (approx). But will you have a computer with an ATA connection or SATA that can support those drives in 8 years? Who knows...

This same logic can be applied to any current technology. DVD-R will get obsoleted someday, will the next gen players be able to read them? etc etc

http://planetskydive.net/ - An online aggregation of skydiver's blogs.

[Fuzzy 0]

You really would have to be having a super crappy day to lose two + hard drives at the same time, especially if they are in different locations.

One advantage to using external drives is when copying everything it's faster than when having to access and copy multiple discs.

Long term in the digital world is much different from that of the traditional archiving of old prints and negatives where the originals cannot be replaced.

With digital media it’s fast, easy, and relatively inexpensive to make exact true copies, but it really is important to keep multiple copies, and know how to find them....



Thanks

Fuzzy
Ambition / Ability: Know the difference.

[docjohn 0]

What about the lifespan of miniDV tapes? I have miniDV tapes going back to 1999.

Tranferring them to DVD would take 13 gigs per tape (hour long tapes, 3.6 MB per second)

Thoughts?

Doc
http://www.manifestmaster.com/video

[DSE 3]

Archive grade DVDs are supposed to have a 100 year shelf (riiiighhhhtttt). HDD will last if you mount them occasionally. Tape, if kept dry, will last forever as far as we know, but you need a device on which they can be played back. BD discs allegedly have a 100 year life span too...

[Fling 0]

There are several issues wrapped together here.

1. protecting from user (human) and software errors,
2. protecting from mechanical failures,
3. protecting from "technological shift" problems.

External drives, or any writable medium while mounted, can suffer from a "whoops, I didn't mean to...", or an application or OS can corrupt/erase files. Type one errors. They also suffer from type two errors such as a hard drive failure (and a power supply problem *could* take out two drives simultaneously) or getting your camera's CF card whacked whilst enjoying a malfunction/reserve deployment for example.
Optical discs (Write Once, Read Many) once written need to be protected from light, stored upright, low humidity etc. Assuming good quality discs to start with, another thing to consider could be to use both -R and +R since they utilize different error correction coding schemes. Test that they can be read using a different machine (drive).

Tapes (DDS or LTO for example) are physically robust and do have a fairly long proven lifespan. They also are much more expensive. On the up side they also tend to have much longer product cycles and backwards compatability designed in. For example, DDS type drives can usually read two generations back so they can reduce the type three problem a bit.

Solving problems one and two can be done. There is an industry built on it, but ease of use and reliability are definitely a function of cost.

Oops, I can see I'm going on... This is a problem I've been struggling with for 20+ years of raw experimental data so I worry about it. Problem three is, and will continue to be, quite a problem. As suggested above a "rolling" backup renewal system is probably necessary.

Bottom line: The optimal reliability/cost may be DVD-R & DVD+R, two copies of each (one offsite). Very important: make sure you can read back what you have written!
Don't assume anything. Test your backups after making them, then periodically there after. All four copies shouldn't fail at the same rate and therefore you get a shot at recovery. I would stay away from external harddrives since they contain both the data and the read/write mechanism. Separate your data from the device: it is safer and your capacity is then limited only by the size and number of media units (discs or tapes).

One last thought. Consider the file format as well. Is it open? Will there be software to read/process it in Twenty years?

Digital: so many possibilities, so many new problems

Alan

[Fuzzy 0]

Hey Al

In reference to

Quote

getting your camera's CF card whacked whilst enjoying a malfunction.....

Recovery software is pretty cool – managed to recover the pictures of a certain jumper’s wife’s tandem (your's in case you were wondering
The digital storage topic definitely doesn't seem cut and dry. The articles, links and all of the comments provided forwarded by everyone here have given me lots of info for the talk at our meeting. We’ll more than likely have a problem fitting it all in in the time allotted but that’s a good problem. Still happy to hear more coments though.

I think a good part of the session will be to highlight the various options, listing the pro’s and con’s of each, and risk management.

Fuzzy
Ambition / Ability: Know the difference.

[PhreeZone 14]

Something else to consider: http://www.adobe.com/products/dng/index.html Most of its just pushing DNG but it talks about support for formats and hardware also.

Yesterday is history
And tomorrow is a mystery

Parachutemanuals.com

[DSE 3]

DNG, DPX J2K are all great (and powerful OZ) formats for the future, no doubt.

[cpoxon 0]

I wonder if online storage will be viable/cost effective?

I currently use 5GB of free space on SkyDrive to store all my personal favourite pictures. GMail currently has 7GB of storage and Google Pictures has 1GB for free. There are lots of online storage services that use Amazon S3 services which are pretty cheap. Amazon S3 (write it yourself) charges 15 cents per GB of storage and JungleDisk, one of the consumer services based on it charges the same amount. The underlying services are based on massive redundant arrays in multiple datacenters so you only need to upload it once and it should be safe. So you can have 100 GB of storage for $15 a month. How much is spent on buying/renewing hard drives and media? One downside is if Amazon does titsup, the data could be lost, but that's fairly unlikey.

So are free/paid services a viable alternative to local physical media?

Skydiving Fatalities - Cease not to learn 'til thou cease to live

[Fuzzy 0]

Not familiar at all with on line storage but it sounds interesting.

Are there any, or what are the security issues that would need to be considered?

I understand that there really is no 100% guarantee, but if a professional photographer stored their originals in a fashion such as this what are the chances of someone else gaining access to them.

Fuzzy
Ambition / Ability: Know the difference.

[cpoxon 0]

Quote

Are there any, or what are the security issues that would need to be considered?

I understand that there really is no 100% guarantee, but if a professional photographer stored their originals in a fashion such as this what are the chances of someone else gaining access to them.

Minimal. All files are encrypted (on the S3 based systems at least) and all services require username/passwords to log in and files are generally not publically available unless specifically shared.

Skydiving Fatalities - Cease not to learn 'til thou cease to live

[cpoxon 0]

Quote

I wonder if online storage will be viable/cost effective?

I currently use 5GB of free space on SkyDrive to store all my personal favourite pictures.

...

So are free/paid services a viable alternative to local physical media?

Skydrive have upped their free storage space to 25 GB.

Skydiving Fatalities - Cease not to learn 'til thou cease to live

[PhreeZone 14]

Quote

So are free/paid services a viable alternative to local physical media?

On this one... no. You don't need to look past the issues of Digital Railroad this summer and you will see that even pay hosting companies are not that great of an option: http://www.diablomanagement.com/announcements/digital-railroad-inc.html DRR gave their customers less then 24 hours notice that the services were going offline forever. http://www.nppa.org/news_and_events/news/2008/10/digitalrailroad.html

Yahoo closed up some of its photo galleries in the last year or two also so even having millions in quarterly revenue is not enough to keep the disks spinning at some places.

Yesterday is history
And tomorrow is a mystery

Parachutemanuals.com

[howardwhite 3]

I use Mozy. It's owned by EMC, one of the big players in storage, so is probably not going to run out of cash suddenly.
And I have recovered a fair amount of stuff that had been on an crashed external drive.
That said, the Mac client version is cranky at best, and maybe buggy.

HW