Thursday, June 5, 2014

Preventing Bit Rot, Part 2

Boy, did I get a lot of email from last month’s blog!  Most people had no idea that even though they were making regular backups and they did a file system check on a regular basis, their valuable files were STILL vulnerable to corruption.  It’s rare that anything I write leads other people to action, but it happened in this case with Patrick Corrigan, author of the book Data Protection for Photographers, created a blog post which essentially mirrored my own conclusions.  You can read his scholarly take on the issue here.

So I've learned that every time I hit a nerve like this, it makes sense to delve deeper.  One of the best suggestions to come out of the blog comments was the reinforcement of the idea to use a more advanced file system like ZFS (Unix) or ReFS (Microsoft Windows 8.1, a compelling reason to upgrade right there!) which is designed to combat this very problem.  One of the most cost-effective and easiest ways for a technically-inclined person to do this is to download and install FreeNAS onto some old computer hardware and have that act as your server.  It's free (except for the hardware), and your data gets the extra level of protection it deserves without you having to switch computers or operating systems.

Archival storage options

Let's talk a little more about archival storage.  CDs and DVDs are NOT the way to go, as they have an AVERAGE data life of 3-5 years.  That's why your CD's from the 80's probably don't play too well anymore.  This kind of storage impermanence is greatly concerning the Library of Congress, who is currently researching the issue - in fact, they are actively seeking your donations of CDs and DVDs that don't play anymore for autopsies.  Here's a succinct problem statement from the Atlantic magazine.

Once upon a time the best option out there was a special CD (now, DVD too) whose reflective layer was made out of Gold.  Initially made by Kodak and now offered by Verbatim (among others), they claim 100 year permanence but this article says that they might not be readable in every DVD reader because the gold layer is not as reflective as the easily-degrading silver.  What's better?  The Taiyo-Yuden / JVC Thermal-Lacquer DVD which has the highest rated compatibility and is said to last about 80 years.

Then there's MDISC, the company which claims to have 1,000 year storage life for DVD and Blu-Ray discs and is readable everywhere.  It uses a specially modified writer like this one, and the platters cost about $3 each for DVD and $5 each for Blu-Ray.  Not bad.

Anyone who starts driveling about “Yes, but will there be readers around 1,000 years from now?” is asking the wrong question.  After reading recent accounts of the levels of effort required to recover Andy Warhol’s digital art from his old Amiga floppy disks, or the massive undertaking to resurrect the original Earthrise image from old NASA tapes, one quickly realizes that you don't have to wait 1,000 years (or even 50 years) for things to become unreadable.  I may actually invest in this because I'm under the illusion that my stuff is worth keeping.  Also, my father is also very concerned about the longevity of family movies.  (Little does he know that the original film format will last much, much longer than either the VHS copy or the subsequent DVD incarnation.  Oh, the irony!)

Next time in Cameracraft

In the next Cameracraft magazine, I show how I was able to change the color of the background from blue to orange using this one tiny filter (and no Photoshop).  There's a lot of other good stuff too, which I'll talk more about next month.  If you're tired of just reading about the latest equipment and are looking to be inspired by people who are doing important things with their photography, then support this old-school anti-establishment holdout and subscribe to Cameracraft today!

Ebooks on the Horizon

Here's what's in the pipeline:
  • Sony Alpha 6000 co-authored by Ross Warner
  • Olympus E-M1 
  • Fujifilm XT-1 by Tony Phillips
  • (Sony A77 II and RX-100 MK3 are planned as well)
As always, if you're interested in being notified when these are available drop me a line at Gary at Friedman Archives dot com.

Seminar in September

Surely you've heard of the Friedman Archives Seminars by now!  Heralded as being the anti-DPreview by newbies and experienced shooters alike, these enjoyable 2-day seminars teach you the long-lost secrets of the Kodachrome shooters, who knew how to get "Wow!" shots without needing Photoshop (or even a really fancy camera).  Light, composition, creativity, seeing, and other things you wouldn't think are teachable are covered in Day 1, with the technical stuff everyone wants to know covered in an intuitive way in Day 2.  Yes, my books will teach you what every feature of  your camera does, but the seminars drive home what's really important to great, high-impact photography!

Although the seminars are officially on hiatus this year, I'll be holding a special one this September in Southern California.  Details are still pending but it will be a special event. Drop me a line if you'd like to attend and let me know where you'll be coming from!

Another Grandchild

I suppose I should start talking about photography again. :-)  A recent trip took me to New York for a memorial service and then to Boston to await the arrival of Grandchild #4.  I packed my portable studio with me (three wireless flashes plus a bounce card - a large white sheet was waiting there for me) and I produced a few keepers. :-)  All of these were taken with the A7 and either the kit lens or the Minolta 50mm f/1.8 with LA-EA4 adapter.  And since the all-white or mostly-black backgrounds would have thrown off the exposure meter, all the flashes were set to manual output (usually between 1/4 and 1/8th of full output).  The chapter on wireless flash in most of my books discuss how you can get shots like these, too!

Two flashes on floor, pointing up.  One had some spill that just covered her face for that "Mommy Glow" look.
Wide Angle
This one's been training to be a Mommy (and a Red Sox fan) ever since she was born.
Shirley Temple reincarnated, with two gridded flashes - one on the camera left (illuminating the front), one behind her (illuminating just the hair.)  That's not Instagram-induced vignetting you see there, that's the real light falloff from the gridded flash.  It took me 10 minutes to set it up but I only had 40 seconds to shoot before she lost interest.  These are the same grids that were introduced in this blog post in 2011.

Until next time,
Yours Truly, Gary Friedman


  1. The trade-off for using something like ReFS is compatibility, especially for a portable drive - most of us probably have a mix of PC's and OS's.

    Also, I prefer NAS4Free over FreeNAS; it's more of a continuation of the original FreeNAS than the current version of FreeNAS.

    A very important point for both: if you are using ZFS and really care about bit rot, use a motherboard with ECC memory! Although there's debate about exactly how critical this is (I'm currently using old hardware, but when I build a new NAS4Free system, I will use ECC memory), it's something to consider. A good post on this topic is

    Also, remember what the NAS4Free guys say: NAS is not backup! You still need to backup your NAS.

    Finally, Virtual Machines can help with old software; I'm using VirtualBox to run old software that won't run on Win7/8.

    1. Outstanding points!! Thank you for sharing. GF

    2. You know, upon further thought, the problem of [ReFS is not yet universally readable] can be solved by keeping them only on the main drives, and the off-site backups can still be NTFS. We don't care about bit flips on the backups, right, since they will (hopefully) be written over once ZFS fixes it? GF

    3. I was thinking along those lines (fixed drive to ReFS), too, but in my case, only my wimpy (but very useful) tablet is Win8 -- and I'm not sure I'm ready to move on from trusty and tried NTFS. And if Windows crashes and you have to do maintenance, NTFS is probably easier to work on.

      Another approach would be to flip it: store the master images on the NAS, and just use the PC for working images. However, I'm sure there are disadvantages to this approach (probably speed, and extra hassles).

      And, of course, another backup solution is to use a compact NAS to backup your main NAS (there are some pretty cute mini-ITX NAS cases from Chenbro and CFI).

      Finally, there is debate over the best NAS drive setup: mirrored, various numbers of parity drives, etc. IIRC, many people say only 1 parity drive is dangerous.

      Right now I'm doing mirrored (dual 1.5T drives), but later I might go to Z2 (5 drives, 2 of which are parity drives). This stuff can get so complicated - another approach is to buy a Drobo, but I prefer to know what's under the hood.

      --Tony T (7D owner - and it's still going strong)

  2. Gary, great post as usual. Questions:

    1. Any idea on when the new a77ii book will be available?
    2. Will this be all new, or a combined book that includes both the a77 and the a77ii material?

    thanks again.. Mike

    1. I still don't have an A77II in my hands, so it can't start yet. And let's face it, very little has changed. It will be a new book with new formatting with the goal of having it look better on e-readers. GF

  3. If you use DNG, hashes of the image data (but not the metadata) are embedded into the file as of DNG 1.2. Lightroom 5 added the ability to verify these hashes.

    Regular verification of your collection combined with a good backup strategy (at least two dissimilar systems, one off site) should provide decent protection against loss or bitrot for most people.

    1. What does lightroom do if it detects that the image hash does not match the image? Does it just flag it or can it attempt to repair the image as well?

  4. Hi Gary, You may not be aware but MDISC was largely silently changed a few years ago so the media being manufactured today is no longer MDISC but rather plain BD discs. They are still trademarked under the same name but it no longer offers that guarantee, and media is largely no longer available.

    1. Hi. I wasn't aware of that; thank you for letting me know. I personally thought they would just go out of business due to lack of demand; changing the product to eliminate their compelling reason to exist isn't something I would have predicted.


Thank you for your comment! All comments must be approved by a moderator before they will appear.