Computational Photography

Editor's note: This topic is so important that in 2022 I was asked to give a zoom lecture to the Royal Photographic Society in the UK.  If you prefer watching video to reading a long blog post, then you may wish to absorb the information this way:

Computational Photography

This blog post has many beginnings...

Beginning #1

I get many emails from photographers the world over, expressing frustration that they schlep their high-quality equipment, shoot RAW and post process, all the while their significant other shoots a similar image with their iPhone, and then posts it to Facebook seconds after it was taken - and the image looks great, with no post-processing needed.  How humiliating!

Beginning #2

In 1973, Paul Simon wrote a song called "Kodachrome", which he said "...gives you those nice bright colors, give us the greens of summers, makes you think all the world's a sunny day".  According to Wikipedia, "... the real significance was that Kodachrome film gave unrealistic color saturation. Pictures taken on a dull day looked as if they were taken on a sunny day. (To correct this, serious photographers would use a Wratten 2b UV filter to normalize the images.)"

Years later, Fujifilm would produce films that made Kodachrome colors look subdued by comparison.

Today, smartphone images represent the latest in a trend to create people-pleasing images that deviate from how the world actually looks to a raw sensor.  Is it still photography with so much misrepresentation going on?

Beginning #3

When the Light L16 camera first came out, I thought it was genius and I thought that this would be the future of smartphone cameras.  This flat slab of a camera employed 16 small sensors/lenses of various focal lengths and stitched several of them together to create a high-resolution 52 MP image better than what any single sensor could produce.  Different focal lengths were combined to emulate a "zoom" between the fixed focal lengths.  The camera was able to produce a depth map by configuring at least two of the lenses into a stereo arrangement.  You could change the depth-of-field after the fact.  If there was ever a good example of what Computational Photography can achieve, this was it - produce an image of greater quality than just what a sensor and optics can provide.

As great as the idea was, plastic optics, a slow processor, sluggish desktop software, and a high price doomed the first iteration.  The company wisely regrouped and focused (no pun intended) on licensing their technology to smartphone companies, resulting in the 5-camera Nokia 9.  Unsuccessful in the marketplace, the idea died.  

Beginning #4

When 35mm film first came out, the "serious" photographers shunned it, as it offered an inferior quality to the medium-format films being used at the time.  Eventually, convenience won out, as people decided the quality was more than good enough for their needs.

Beginning #5 - Why can't the camera just make it look the way I see it?

In my seminars, I would talk about how the camera and the eye see light differently.  I explain to attendees that the limited dynamic range of our modern sensors is narrow on purpose.  I then show this "devil's advocate" example:

This image was a merged bracketed exposure - perhaps 30 stops in total range; much wider than what the traditional HDR feature on your camera can produce.  It shows everything my eye could see from the detail in the backyard through the doors, to the detail in the shadow under the piano bench.  

But an image that can see everything your eyes can see can look very flat and low contrast, as in the example above.  "One day", I would say to my seminar attendees, "psychologists will figure out what kind of image processing is happening inside our brains, and then the camera would just make it look like it appeared to our eyes."

===

My friends, that day has nearly arrived.  And the advancements didn't come from the camera companies.  It came from the smartphone manufacturers who had to be clever in order to achieve higher quality results than what their camera's tiny lenses and sensors would otherwise allow.  Yes, the iPhone images can look relatively poor when you pixel peep, and the saturation and HDR might be a little over-the-top when compared to a traditional camera, but if all you do is post to Instagram that difference become meaningless - people LIKE those nice bright colors, and those enhanced greens of summer.  Plus, in my experience, most modern smartphones handle difficult light and HDR much better / more naturally than shooting in HDR mode, and just as good as spending two minutes tweaking the RAW file with conventional cameras to make it look the way your eyes saw it.

What computational tricks are the smartphones using that conventional cameras aren't?  Is it really photography when so much manipulation is automatically applied, or when the image is enhanced to the point of near-fiction?  

Computational Photography

The idea of merging images from multiple sensors (as in the Light L16 above) is just one example of computational photography.  There are many other techniques that smartphones employ that traditional cameras just don't.  Below are just a few examples (as always, click on any image to view larger and sharper):

How the Samsung Galaxy S21 analyzes your image and optimizes faces.  This unfortunate example was taken from their website, as the claimed "improvement" is hard to see.

Smartphones are incorporating machine learning techniques (sometimes coupled with dedicated neural processing hardware) to identify subjects and automatically perform the kind of editing that would brighten the image, smooth the skin, fill in detail, and blur out the background in certain modes.  DSLRs often had some of these features scattered throughout their menus but rarely would they be automatically invoked without the photographer's knowledge.  Smartphones do it every day.  Normal people love the results they provide straight-out-of-camera.

Here's a straight-out-of-phone example taken in "Photo" mode.

Scroll down to see the same picture taken in "Pro" mode, which doesn't try to enhance the image.  (I'm separating them on purpose).

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. (keep scrolling)

.

.

.

.

.

.

.

.

.

.

.

.

.

Taken with the smartphone in "Pro" mode.

Why didn't I show you these examples side-by-side?  Because if I had, you would have looked at the first image and shouted, "Overprocessed!"  Here, I'll prove it:

See?  You probably didn't object as strongly when you saw just the one image.

Comparisons aside, the new algorithms also provide the most realistic HDR I've seen without resorting to bracketing and tone mapping.  Keep in mind that the images were also designed to be viewed by a high-contrast display typically found on smartphones, boosting the bright-and-saturated look even further.

Here are some more impressive examples I took with a recently acquired Samsung S21 (not the 'plus' or 'ultra') in both "Photo" (left) and "Pro" (right) mode.  (Pretend the buildings are straight :-) ): 

"Photo" mode

"Pro" mode

Photoshopped .dng file

More examples of how smartphones handle High Dynamic Range:

A conventional camera couldn't do this without shooting RAW and postprocessing.

The Merging Several Adjacent Images Technique

Another computational photography trick that the big cameras tackled first was the idea of merging several shorter-exposure pictures into one, simulating a longer exposure.  Traditionally, really long exposures were enabled by using dark Neutral Density filters; then programs like Photoshop and StarStax came out allowing you to do the merging on your computer, while averaging away the random noise in the process.  Sony's Smooth Reflections app did the same thing in-camera, which I vastly preferred to using ND filters to get long waterfall shots.  (I compare the two techniques in this blog post from a few years ago.)

A waterfall in Iceland.  It was too cold and the constant hail discouraged me from using ND filters; Sony's Smooth Reflections App in the A7R II did it in a fraction of the time and it nailed the exposure on the first try.  Sony cameras have since stopped supporting downloadable apps, so that's why I will always take the A7R II when I travel. 

The Shaky Hand Technique 

Wait, it gets better!  You know how some cameras offer a feature called pixel-shift, where you put your camera on a rock-solid tripod, and the camera takes multiple pictures, shifting the sensor a quarter-pixel in each direction, and then merges these into one über-high resolution image?  Smartphones can emulate that technique, not by moving the sensor a quarter pixel, but by utilizing the shakiness of your hand.  Smartphones can continuously capture and buffer images at video speeds, allowing the camera to select the best source images - that is, those which are offset about one pixel from the other (and also not blurry) to use for this.  All invisible to you.  A higher-quality image (in some cases also a higher resolution image) without the burden of higher-quality hardware!

This technique also the way many smartphones do image stabilization.  Well, that's not true - when you are composing your image on your phone, the phone is capturing full-res still images at video speeds, and analyzing each image for sharpness.  When you press the virtual shutter release button, the camera goes back through the buffer and simply selects the most recent image that is sharp.  Genius, but that also means you won't necessarily get that 'decisive moment' you were hoping for.

Another great benefit to using the 'shaky hand' technique is you can get also get a real value for each Red, Green, and Blue pixel instead of relying on Bayer demosaicing technique.  (Your next question will be, "Will I notice a difference?"  The truthful answer is, "Probably not", although the image noise has the potential to be lower.)

Night Mode - Without a Tripod

Combining multiple short-exposure images is also how the various "Night Mode" features work, and it's a combination of all the techniques mentioned above: shaky hand, multiple image captures, in-phone alignment, and averaging away all the noise.  In testing out this feature I compared it to what my Sony RX100 VII produced (which technically needed a tripod since the SteadyShot would only do so much in extremely low light).  

S21                                       RX100 VII

S21                                 RX100 VII

  

S21 in "Night" mode

S21 in "Pro" mode (this is actually how it looked to my eyes - it was DARK there!!!)

Brightened DNG file (not as good)

And here are a few more Night mode images to share, just because I like them :-)  :

  

  

Remember, these are all snapshots.  But how nice to get them all to look better without having to post-process!

S21 TIP: If you're shooting in PRO mode you have the option to also save the image as a .dng (which is kind of like a non-proprietary RAW file).  If you also have your images automatically uploaded to Google Photos, you should know that if you have the Upload Size set to "Storage Saver",  your raw files will be converted to .jpg before being uploaded, losing all the benefits of shooting RAW.  Instead, hook up your phone to your computer via USB and download them that way.  Quality loss avoided.

Time-Of-Flight

Then there's everyone's favorite "Portrait" mode, which simulates the kind of shot you'd get with one of those expensive white 70-200 f/2.8 portrait lenses.  The way it's done is pretty complex - the camera has to figure out which compositional elements are close and which are far, and it will just do a nice Photoshop-like Gaussian blur on the objects that are far.  How the phone determines the distance varies - some phones like the latest iPhones use laser-based LiDar to build a 3D model of the scene; other phones do the same thing with IR light (using a sensor called Time-of-Flight) - both techniques send out either a laser or IR light and time how long it takes for the light to reflect back off the subject.  The longer it takes, the further away the subject is.  Still other phones like the early Google Pixels had to be clever about building their depth maps without any such special hardware, leaning heavily on machine learning and Convolutional Neural Networks to identify what the subjects might be.  Clever but not as good, which is why early Portrait Mode examples were kind of sloppy around the edges; sometimes stray hairs or parts of clothing would be blurred which kind of gave it away.  I'm not seeing those artifacts now, though.  

Surely you've all seen portrait mode examples before.

Here's an image that a conventional camera couldn't take.  The phone identified four faces and blurred everything but the faces.  So you end up with the front row that's in focus, the space between the front and back rows that are blurred, and the back row that's in focus.  Good or bad?  You decide. 

I love the Portrait feature; when coupled with the phone's optical telephoto lens it produces nice results and saves the weight and expense of a 'real' lens (although people won't take you seriously as a wedding photographer using this :-) ). 

TIP: Normally the depth map can't be stored in .jpg format (nor RAW format for that matter), which is one reason the new HEIF file format is now a thing.  In addition to higher quality compression, there's also a means of storing the depth map information inside.  (Also video, audio, and a few other things.)  Why is that useful?  It's possible to open the depth map in Photoshop as a variable density selection layer, allowing you to control your own blur - the "redder" the mask, the more the blur is applied.  

What I don't understand is why can't modern cameras with > 600 focusing points be programmed to generate a depth map automatically when a picture is taken?  Just evaluate the distance behind each AF point (the camera does this anyway when identifying the closest object to focus on) and build a depth table?  I would love to be able to use my RX10 IV at 600mm and then get higher-end-camera blur after the fact using this technique.  Something for Sony to consider for their next models.

Lighting as an Afterthought

That depth map alluded to above also allows for after-the-fact CGI lighting for your portraits (which I proposed more than a decade ago) which Google is starting to offer via the Google Photos app on images it thinks are workable.  While the purists will insist that "there's no substitute for good light", the rest of the world will say, "Ooooh, another Instagram filter!" 

Image courtesy Google's blog.  If it's not animated for you, have a look at the original here.

Machine Learning

No overview of computational photography would be complete without discussing machine learning, where you feed a gazillion images to an algorithm and let it learn what common things are supposed to look like.  Why would that be helpful?

I'll just cut to the chase here.  A few months ago Google published some amazing examples of their Machine Learning projects, in this case to scale very-low-res images into high-res ones.  The results contain detail that wasn't there originally - detail that was extracted and transmorphed from the training images.  Have a look at some closeups from their recent published paper:

Here, the reference image in the right column was the starting point.  A low-res version of that image (left column) was fed into their SR3 algorithm, producing the center column which is amazingly close to the original reference (impressive until you start to look closely).

Here is a close-up example.  Notice that the eye looks convincing but the texture in the hair does not.

Another case where the detail (the hair) looks convincing, yet also completely different from the original.  The algorithm simply makes up convincing-looking detail.

Here's the acid test.  Can it reconstruct letters or candy wrappers?  No.  It wouldn't do well on license plates either.

Google also used a second mind-blowing technique, which builds the high resolution image from pure noise.  On what principle does THAT work?  Google explains all in their easy-to-understand online paper.

Right away two things are clear:

1) These techniques won't work everywhere; the best results are obtained when the test shot is similar to an image from the training set.

2) This kind of technology might be great for smartphone snapshooters, but absolutely wrong for things like surveillance cameras and video, since the detail presented wasn't necessarily there in the original scene.  (And of course you know that's EXACTLY how law enforcement is going to start using it until the lawsuits start.)

The Pixel 6 phone is going to be formally announced in a few weeks; how much do you want to bet that some version of this will be used to get great high-res images from a standard smartphone camera module?

===

So why am I telling you all this?  As alluded to earlier, my old smartphone died and I got myself a Samsung Galaxy S21.  And while there's no contest when it comes to pixel-peeping and comparing to conventional cameras, I continue to be impressed by what the camera does on its own, behaving like how we wished point-and-shoots would have worked since the early days of point-and-shoots.  They produce more people-pleasing images out-of-the-gate than your conventional camera.  Which leads directly to Beginning #1 above.

We live in amazing times.  The gap between smartphones and traditional cameras continues to shrink.   I even find myself leaving my RX100 series cameras behind whenever I go out now, something I never would do five years ago.  Ever since I was able to license images taken with my older Galaxy S8, I stopped worrying about enlargability.  The market demand for high-res images has dropped considerably over the last 20 years, and companies like Adobe and Topaz are developing image-scaling tools that are decent (as long as you don't examine your images with an electron microscope).

(I'll still be using conventional cameras in the new studio, though!)

In the Pipeline

The ebook for the Sony A1 is being translated into Spanish.  Email me (Gary at Friedman Archives dot com) to be notified of its release!  

Also, version 1.04 of said A1 ebook has just been released.  You should have automatically received a free update, but if not just email me your purchase receipt and I'll provide you with a download link.

Next Time in Cameracraft

I sit down for a conversation with Andrea Pizzini, who was so frustrated by the misinformation regarding the COVID 19 pandemic that he spent the last six months documenting a COVID ward and how the virus has personally affected real people he grew up with in his native Italy.  

Zoom Lecture for your Photo Club

I've been giving Zoom lectures to photo clubs around the world for over nine months now, and I've gotten (that's a word!) the setup down to a small footprint, that can be used in small spaces (like a kitchen table in a tiny 2-bedroom condo in Boston :-) ).  In the image above I've highlighted five key items in my setup:

1. A camera that's way better than a webcam.  Here it's the RX100 V, at eye level so attendees aren't looking up my nose.
2. The screen so I can see all of the participants; see if anyone's raised their hands or are sharing something.  (This is important to me, as I like my lectures to be interactive and being able to see people is paramount and more enjoyable.)
3. A camera attached via HDMI so I can give a live demonstration of an exposure or wireless flash technique or camera operation principle.
4. My "control panel" where I can switch between different virtual cameras (the RX100, the demo camera, my laptop screen, a video, or the powerpoint program running in the background.)
5. An HDMI monitor so I can see what the participants see (usually a powerpoint slide or the demo cam).

(This is a more portable version than my original setup from last year, which I blogged about here.)  

I've already lost track of how many photo clubs that have hired me, but I can tell you this: Every club has enjoyed them immensely - some have even asked me back to give a 2nd or 3rd lecture on different topics.  

I can do this for your photo club as well!  My most popular Zoom talks have been:

• RAW vs. JPG – I tackle this very religious technical subject with clarity and challenge the experienced photographer to re-think everything they were told was true about .jpgs. (This is, by far, my most popular talk and the one that has changed the most minds about what is true and what is hype.)
• How to “Wow!” with Wireless Flash – Here I demonstrate how easy it is to move your flash off your camera and add great drama with no need for technical knowledge. Think a new lens will improve your photography? Learning to use light will have a dramatically greater impact on your images.
• The Forgotten Secrets of the Kodachrome Shooters – How pros in the 1960’s got “Wow!” shots without fancy cameras and without Photoshop. (These secrets apply to today’s digital cameras, too!)

I can also put together a talk to address your club’s most pressing questions. (Hey, I’m working for YOU!!)  Contact me at Gary at Friedman Archives dot com for more details.

That's it for now!  Until next time,

Yours Truly, Gary Friedman

(Creator of the densest blogs on the planet (tm))

ZZ Top, the latest in my year-long Quarantine Beard Self-Portrait series