Tuesday, March 24, 2020

I invented a camera whose output could be authenticated. Nikon and Canon stole the idea. What happened next will shock you.

"Deepfakes" have been around for years, like this Coca-Cola commercial mixing living and dead celebrities.  But how can you prove if a video hasn't been manipulated?

(Note: This is an expanded version of the article I wrote for Cameracraft magazine.  I'm sharing it now because the idea needs to be out there.  Plus, I was on tap to give a TEDx talk about this subject in April 2020, but of course it was cancelled due to pandemic concerns.  So I'm putting it out there.  The world needs this invention!)

Once upon a time there was a saying: “The photograph doesn’t lie”.  While mostly true, you could still lie in the old days by attaching false captions or using a forced perspective.  Lying by manipulation came much later -- it was used heavily by the Soviets during the time of Stalin, and then by the advertising industry (which is synonymous with lying, really) with the invention of the Scitex imaging workstation in the 1970’s.  But the ability to really lie via manipulation didn’t reach the masses until Photoshop came along. 

1989.  US Ambassador Vernon Walters presents photographic evidence at the United Nations Security Council, supporting his claim that a Libyan MiG-23 shot down by US fighters had been armed.  “It is untrue!” exclaimed the Libyan ambassador.  “The pictures were fabricated; they were directed in the Hollywood manner!”
People (journalists and academia, mostly) started to sound the alarm in the 1970s about how you can’t rely on the photograph for evidentiary purposes anymore.  And I started to collect examples of famous photo manipulations in history which made a difference (good and bad).  Some historic examples are sprinkled throughout this article.

Back in my NASA days I identified this as a problem that needed solving – in my view, society was relying too heavily on the image whose sanctity was eroding, and I made it my mission to restore it.

Photographic proof that Rambo and Groucho were at the Yalta conference.
My Solution

The solution I came up with imposed no restrictions on image use and required no special or restrictive formats.  Taking a picture with my special digital camera would produce two files: The first is an industry-standard image file like a .jpg or RAW file, which can be used as you would any other image file (you can even manipulate it if you wish).  The second file is a smaller encrypted “hash” file called a digital signature which is kept alongside the first file to enable validation later on. 

If the image hasn’t been manipulated, you’ll be able to prove it in court by feeding the image in question, the encrypted hash file, and the camera’s serial number into freely available verification software.  If even a single bit has been altered, the validation will fail. 


The comic strip Bloom County was one of the first to call this out as a problem in the 1990's.

Wait, there’s more!  My invention also sought to thwart the classic, non-manipulation methods of lying as well, such as attaching false captions or taking a picture of a manipulated image.  To combat these time-honored methods, my camera would add a colorful border around each image which contained bitmapped text describing when and where the camera was: It would note all the usual data covered in EXIF fields (shutter speed, f/stop, ISO, white balance), the time and date, a compass showing what direction the camera was pointed to, the distance the lens was focused to (to catch copying a manipulated image), and even the GPS co-ordinates of where the camera was using a built-in GPS receiver.  (Clearly this was an optimistic prediction, because back in the 1990’s when this invention was proposed, GPS receivers were so big they took up a goodly portion of the B2 bomber’s electronics space.)

Another time-honored way of lying with the image; no manipulation required.  My system would help thwart that technique as well.

How the camera works would take too long to describe in this short blog post; however if you’d like to learn the details, I refer you to read the full technical paper, published in the Nov. 1993 edition of IEEE Proceedings of Consumer Electronics:
In my proposed system, the environmental variables are authenticated as well as the image itself – in this case information in the colorful border, indicating the time, date, exposure variables, white balance, what direction the camera was pointing, and where you were in the world when the shot was taken.   Here’s an image of youthful me standing next to Captain Kirk, holding another invention – a one-handed keyboard for astronauts.
As a NASA employee, whenever you invent something, NASA owns it whether it was work related or not.  They unleash their army of patent lawyers to file a patent and pursue it until it’s granted.  And they give me a $500 award, which NASA thinks is a great deal because most patents never make any money and the engineer doesn’t have to spend tens of thousands of US Dollars getting the patent on their own.  NASA also has the means to advertise the patent to large U.S.-based companies for licensing, again something individual engineers cannot do.  Fair enough.

The Patent

There was interest.   Polaroid Corporation, who had been searching for a solution to this very problem for over a decade, flew a cadre of executives to California to hear me give a talk on the subject, with the intent of possibly licensing the idea.  But Polaroid collapsed before it could be acted upon.
Polaroid was extremely interested in licensing the technology; they even flew a cadre of high-level managers out to hear me give a presentation.  I threw this prototype magazine ad together, thinking past the sale: "Here’s what you’ll be able to communicate to the world!"

Then something interesting happened.  In 2003, Canon stole my idea.  With the introduction of the EOS-1Ds, Canon also introduced the Data Verification Kit DVK-E1 which works only with this high-end camera.  Three years later, Nikon did the same thing with the introduction of the Nikon Image Authentication Software, which worked only with their high-end D2Xs DSLR, and later the D2HS, D2X, and D200 cameras. 


Both Canon and Nikon came out with their own version for their high-end digital cameras.

Frustrated, I appealed to the NASA patent office, pointing out the blatant theft of intellectual property, but they did nothing.  As it turns out, the patent was only filed for and granted in the U.S. (Canon and Nikon are both Japanese companies), and besides, it turns out NASA is not in the business of defending IP violations – just filing for patents so they can help justify their existence to Congress.

Just as I was starting to learn to let go of the outcome (a valuable lesson for inventors and non-inventors alike), the whole thing collapsed.  A Russian research firm called Elcomsoft identified a flaw in Canon’s implementation, exposing the private encryption key and rendering all of the cameras using this product inert.  Nikon’s scheme was similarly broken six months later.  In both cases, the secret encryption key had been extracted from the camera, a sign that the hardware system was designed by engineers unfamiliar with data security techniques.  Had either of those companies hired me to be their product manager, I would have made sure that one compromised camera would not have invalided the system for all other cameras.

Scientific American also called this out as being a problem.
I was even more disheartened that neither company made any attempt to fix their product or come out with a new, stronger version.  While Canon made a tiny announcement that the system had been compromised and invited customers to contact them, Nikon was completely silent.  The products just died, just like Kodak’s PhotoCD standard.  And the world either hasn’t noticed or doesn’t seem to miss it.  I interpret this as "The product didn't sell very well", and "Global interest in this problem is weak".

Like all young inventors, I thought my solution was going to save the world, especially in situations where truth is valued, like when the photo was being used as courtroom or insurance evidence.  Shortly after the patent was granted I approached a lawyer friend of mine and excitedly told him about my invention.  He shrugged it off.  “If we ever question the authenticity of an image submitted by the other party, we hire our own photographer and take our own pictures from our own perspective”.   This was an important lesson I would learn time and again as a frustrated inventor – if you have to explain to people that there’s a problem before you can present the solution, you’ve already lost the race to a product's success.

Today we have deepfake videos whose manipulations are getting increasingly harder to detect, making a solution like this even more important than it was in the 1990's when I invented it.  But the patent has since expired, and I'm pretty much done trying to change the world.  I'm sharing this idea in hopes that someone will take it and run with it.  

===
Until next time,
Yours Truly, Gary Friedman


*** UPDATE 10/28/23 ***

The idea lives!  A new standard for image authentication which pretty much copies my idea is being drawn up by the Coalition for Content Provenance and Authenticity (C2PA).  Sony, Leica, Canon, Nikon, Adobe, Arm, BBC, Intel, Microsoft, and Truepic, and Twitter have signed on, and the first camera to employ this "Content Credentials" technology was the Leica M11-P which costs a mere $9,125. :-)   

Of course my patent has long expired, and this adds to a long list of ideas I first thought of back in the 1990's that others are getting rich off of now.


15 comments:

  1. Interesting post. Perhaps as a society, we no longer value the truth?

    ReplyDelete
    Replies
    1. In the courtroom and in science in general, we still value truth. Elsewhere, not so much nowadays.

      Delete
    2. Not really, Gary: courtrooms' main function is to decide who wins, and by extension, who loses; this determines guilt (and financial obligations) and the not guilty. 'Truth', in the majority of cases I am familiar with, is a casualty, not an outcome.

      Delete
    3. Well, Kit, that makes me really depressed.

      Delete
    4. Sorry, Gary; as someone with a science background myself, I find how the 'game' is played in courtrooms is depressing. Like WOPR in War Games, the conclusion that anyone who thinks about these things is "the only winning move is not to play."

      Personally, I value the truth as a concept, a way to live, and an orienting principle. We all get to decide on these matters ourselves. Anyhow, way off topic, and I am a fan of your newsletters.

      Delete
    5. Gary, just remember that in every court case there are four parties involved - the accuser, the accuser's lawyer, the defendant and the defendant's lawyer. Two parties will win; two parties will lose.

      The lawyers will never lose...

      Delete
  2. Dear Gary, You are special! Unfortunately, we do not value the truth. Vicki

    ReplyDelete
  3. Hi Gary,
    Gripping story! I belong to Nature Photographers Network (NPN), basically an image sharing and critique group.
    Would you mind if share a link to this article with the group?

    ReplyDelete
  4. Love the article, too bad they didn't see the value of validating images.

    Did not realize before that you were a NASA engineer, I thought you were a photographer at NASA, it must have been a great experience working with such a great group of dedicated experts.

    I recently heard about a high quality image system that NASA used for their NASA rover systems that is now being sold under the name of GIGAPAN EPIC PRO. Is this something your worked with, is it a technology that you think is work working with?

    ReplyDelete
    Replies
    1. Glad you enjoyed the article. You can read more about my time at NASA here: http://friedmanarchives.blogspot.com/2013/06/my-life-as-geek.html (And photos from the Voyager and Galileo data systems here: http://friedmanarchives.blogspot.com/2019/03/nasa-computing-from-1980s.html )

      The cameras used on some of the Mars rovers were 2 MP but they did a lot of mosiacing, like the Gigapan to get hi-res images. More about that in this dpreview article from 2012: https://www.dpreview.com/articles/0353350380/curiosity-interview-with-malin-space-science-systems-mike-ravine

      GF

      Delete
  5. Hi Gary
    I just read you article in Cameracraft about your idea where critography were to be used to stop fraud. That was a good idea but Quantic computation came to my mind. You can now get some relieve from your grief of not having been able to see you idea come to fruition thinking of all the demands you would have had when all those keys you thought were so super secured were now broken and nobody could trust the photos again.
    Or else you can now try to use this new Quantic technology to improve your design. It is true that maybe you are back to design issues due to size and temperature ha ha

    Best regards
    Javier

    ReplyDelete
    Replies
    1. Hi, Javier!

      Thank you for your astute observations. Quantum computing was in its infancy (it still is, actually) back in the 1990's when this was proposed. And when it gets commercially viable, the sanctity of the image will be the very least of our problems. :-)

      As I mentioned at the end of this post, my entrepreneurial days are over. I'll let someone else (maybe you?) investigate the newer methods.

      Glad you're enjoying Cameracraft! GF

      Delete
  6. Hi Gary,
    This was really interesting.
    Many photographers proudly unapologetically post-process so the market for your invention seems really small but you did convince me that your idea has merit, explaining it clearly.
    Thank you!
    Stay safe,
    John

    ReplyDelete
    Replies
    1. John! Great to hear from an old PPC friend. Glad to hear that someone thinks it has merit. :-)

      Delete

Thank you for your comment! All comments must be approved by a moderator before they will appear.