4.23.2012

An observation about the increasing importance of raw conversion software in critical photography.


I loved the way the light looked as it wrapped around handle of this white dish.
Shot at 6,000 by 4,000 pixels.

When I looked at the file at 100% (ISO 400) I could see colorful speckles of grain.

I ran the file through DXO Optics Pro and it handled the noise very nicely. 

I have a confession to make.  I thought cameras had gotten so good that I could just shoot Jpegs and be done with the whole mess of post processing. I know some photographers relish the butt time in front of the computer as much as the rest of us dislike it but, given a choice I’d rather work on a book or  talk a walk my city around and look at the real world.

I recently bought two Sony a77 cameras and, for the most part, I’m happy with the system I bought into.  There’s one thing that bugs me, though; the files have some noise in them, even at ISO’s as low as 400.  Now I’ve read Michael Riechman’s  comments about the noise and the cameras and I know he’s probably right.  We’re too busy looking at this stuff at 100%.  The images at 100% on our screens would represent huge prints and there’s no way we’d be standing so close to them that we’d even be able to see the grain.  But it’s like knowing how they make sausage.  Once you’ve looked at your files at 100% you’ve always got that queasy feeling when you think about big prints and fussy clients.  

And that’s kind of nuts because part of the lure of a 24 megapixel camera is the idea that you’ll be able to print large.  Really large.  In normal sized files I thought the images looked fine and to my clients there was really no difference between what I’d given them, file-wise, from last year’s cameras or the cameras I owned a few years before that.  But the tragic thing was that I would know. Compulsive behavior rears its ugly head.

Here’s what I think is happening.  I think most sensors are noisy little devils on their own and manufacturers pull the images off the sensors and then progressively slather on noise reduction as the sensitivity goes up.  At a certain point you reach a hinge point where you can either have nice detail, peppered with noise, or you can choose a smoother look and sacrifice the impression of finely delineated hair on striking blonde goddesses. 

The most egregious manipulation happens to Jpeg files and it’s horrible because once the camera spits them out they are well nye impossible to fix.  Once they are slurped and greased you can’t un-grease them even with the best software.  Why do manufacturers do this?  I think it’s really a question of how much per camera they want to spend on real time image processing in the camera pipelines.  The finer the control and the tighter the quality integration the more processing speed and buffering you’ll need.  And there’s always a calculus of intersecting value curves that yields the most effective, “I’ll buy it - curve” in the world of marketing.

Sony builds great sensor semiconductors and they are in use in many great cameras. Including famously noise free cameras like the Pentax KR5 and the Nikon D7000.  But they don’t seem to get where the tipping point is on noise and noise artifacts, or what constitutes excessive blurring of the files.  For most people it really doesn’t matter.  It isn’t a life or death issue.  It seems that Sony wants to build in super fast frame rates and big files with skinny buffers and the way they make it all work is with rudimentary “on the fly” noise treatment. Strictly mid-tier.  Nikon and Canon are either putting more effective processors and more complex noise reduction algorithms in their cameras or they have a vat of fairy dust somewhere with which they sprinkle their outbound cameras.  Either way, they leave Sony product in the dust.  Or so I thought...

After reading around the web and revisiting some of the product essays at Luminous Landscape I decided to make an all out effort to make my Sony a77 raw files the very best they could be and to pit them against the high ISO files I’ve accumulated from the Canon 5Dmk2 camera which I owned.  This might seem to be a “Sony-only” blog post but nothing could be further from the truth.  What I’m writing here pertains to a number  of cameras that have gotten a bad rap for high noise (although nothing will save a Kodak DCS 760C at ISO’s over 160....).

I’m going to boil it all down for you.  The way to creamy, dreamy files with good bite and low noise, even at ISO’s like 1600 and maybe even 3200 is to do this:  Turn off the high ISO noise reduction entirely.  All off.  Shoot in raw.  Yes, big, fat raw.  Then bring your files into a conversion program like DXO Optics Pro and handle your noise there.  Or in Capture One.  Even the Sony Image Data Converter program.  You’ll have much more control over the noise reduction protocols and you can offset the reduction in micro fine acutance with adjustments to all the parameters in unsharp masking menu.

I like DXO Optics Pro because, with my Sony a77, shooting in raw, the program will correct for the most common lens distortions, vignetting, chromatic aberrations, known sensor anomalies and other issues all at the same time.  I can override the noise recipes in the program and fine tune each file if I like.  It takes more time and the program is much slower than Lightroom.  I’m upgrading Lightroom as my next step so I can play with the same kinds of controls there and see who have the best combination of speed, convenience and image quality.

What I’m seeing now is that my a77 files are a pretty good match against the Canon 5D2 files up to and including ISO 1600.  The process of running files through DXO is more time consuming up to a point.  But in truth we’re keeping to our basic workflow and only optimizing files in DXO after the client has made a final selection.
  
We may shoot a thousand images in a day or two and after an edit we may be sharing several hundred with our clients.  We ingest the files in LIghtroom 3.6 (currently) and do quick global and “regional” corrections of the images before resizing them and batching them into web res images for online gallery display.  Once the client selects their images I run them through the DXO process and then send the files as Tiffs to PhotoShop for anything that requires selections, layers, or spot color corrections.  And of course, the obvious cloning, healing and retouching.  Starting with a higher quality file that’s geometrically “mended” is a load off my mind.  And it adds a lot of value to my new cameras.  At lower ISO’s they are more than competitive with cameras like the 5D mk3.  

While they probably will never come close to yielding a low noise file at ISO 6400 I rarely shoot at those settings anyway.  It’s a lot more routine to try and shoot at lower ISO’s in order to get the maximum dynamic range. That's why we own various lights.  And at ISO 50-200 the Sony’s are as good as anything but a Nikon D800 for total dynamic range and endless detail.  And at less than half the cost.

The whole point of my blog post today is to point out how critical software can be in grinding out the very best potential images you can get from any camera.  Some camera makers actually make software that shows off the best of their cameras.  Nikon is one that comes to mind.  And some make kludgy software that barely passes muster (like Sony).  If you know where the issues are you can experiment with the best solutions for their resolution.  Whether inside the camera or inside the computer the changes all come down to different software and processing choices.  Sony is betting that most people will find their fast throughput, lower quality solution adequate.  Thank goodness you can choose to take their raw data and make it much better.  If you couldn’t improve on the pictures then no one would bother making and selling the aftermarket processing software.  
In the next few weeks I’ll try to shoot some images that clearly show what I’m talking about here but in the meantime don’t take the limitations for your camera for granted.  The internal software of your camera might be like the cheap tires on an otherwise high performance car.  They’re relatively cheap to replace and you may be in for a much better ride.

Now I’m happy with my cameras.  With the right processing software they do exactly what I want them to do.  

27 comments:

Frank Grygier said...

I use DXO with my Olympus E-5. It is the only software I have found with Olympus lens profiles and sensor optimization. Sometimes the files look so good I swear I have a Canon MII.Heh...

Bill Beebe said...

Let me know what you think of LR4. I'd like to compare notes.

Jan Klier said...

There's always a trade-off between what quality you can achieve in real-time vs. what can be done if time is a little bit more elastic. Of course you can always throw more horse-power at a problem and achieve more and remain real-time, at a significant cost. So it's really a triangle you optimize: cost - response time - quality. As with any triangle optimization, you can have two of the three, but not all three.

Interestingly enough, none of the processes we hold frequently as the quality bar to compare our digital cameras against was real time. Old film processes by definition were time consuming and allowed for proper development. And the hybrid process of film + scanner, which is today's standard for rich and lush image files is in fact also not a real-time process. First there is the film chemistry, and then there is the scanning process, both of which take their own sweet time to come up with a great result.

Based on that observation, shooting raw and using raw converters in your post-production workflow seems to make a lot more sense: allow some more advanced processing to take place at a reasonable cost. With Lightroom the workflow is very seamless, and you almost forget the fact that you're dealing with raw files.

I always shoot raw, partly because it's a non-destructive workflow (as all of my post workflow is), but also because it yields much better results.

FotoEdge said...

With my little E-P3, I feel very comfortable with the Olympus Viewer Software and convert my RAW files through there. Besides the white balance, I sometimes use the format ratio selection and maybe apply a little Art Filters before Saving and moving them into Photoshop CS5.. It goes pretty fast and the results are much richer than the JPEGs.

DGM said...

Thanks for sharing your thoughts and workflow. I sympathize with your customer perception/noise dilemma.

The number one jpeg machine at the moment seems to be the Fuji Xpro1, followed by the Olympus M5. Still, the Sony Alpha system makes more sense for a working pro, I think. The upcoming full frame Sony will be very interesting.... :)

Not being a pro, I just slog along with my little NEX5N and a bunch of classic lenses. Not practical for a pro on a schedule, but it fits my budget.

Speaking of budget, I am currently wrestling with upgrading my old Photoshop CS3 to CS6. I had written Adobe off completely when they cut off the upgrade path. They now seem to have relented and I have till December to choose to upgrade for $199. On the other hand, I could go with Lightroom for $149. Theoretically it would get me the same RAW converter. I understand that they are aimed at different purposes, but I would like to hear your thoughts on Lightroom. It might be worth learning a new interface. After getting the NEX5N I have been converting with RawTherapee. Still learning it. I won't be offended if you ignore this question, I have some practical reasons for not wanting to install the trial version on my very limited resource computer at the moment. Anyone's opinion would be welcome, but not expected.

Craig said...

Pellicle-mirror cameras like the newer Sonys can be expected to produce somewhat noisier images than comparable non-pellicle cameras, since the mirror is diverting some of the light away from the sensor. This is just something that goes with the territory, though I'm a little surprised that it's noticeable as low as ISO 400.

Shooting my 5D Mark II at ISO 3200 almost in the dark taught me to worry less about noise. Yes, there was noise if you looked closely at unreasonable magnification levels (as you point out in this post). But it isn't particularly noticeable otherwise, so I can live with it. I don't think either you or I make images designed to be best appreciated with a 10x loupe, or at a combination of print size and viewer distance that makes it impossible to see the entire image at once.

I am not a big fan of noise reduction, either, because it tends to smear things, as can be seen even in the example you show here -- the edge of the dish looks noticeably softer in the noise-reduced version. Again, this would probably not be noticeable at a lower magnification level, but then, the noise wasn't visible except at 100% either, so what purpose is the noise reduction really serving?

John Ferguson said...

You can also perform two raw conversions of the same file, one favouring detail over noise reduction, and one favouring noise reduction over detail, then blend to taste in photoshop.

kirk tuck said...

Theoretically the mirror should rob only half a stop of performance. I chalk it up to a more primitive noise reduction engine optimized for speed over ultimate quality. I think you can get better noise reduction and still keep fine details with better processing. I agree that it's probably moot at the sizes we generally use stuff but there's an emotional insecurity that must always be dealt with. At least for we mere humans...

pradeep said...

Thank You

The given information is very effective
i will keep updated with the same

industrial automation

David said...

Another good post, thank you. My experience from use of Aperture to process my Canon 400D files is that over the last 3 years the Apple Raw Converter used by Aperture has improved dramatically at processing these files. ISO 800 & 1600 (top end on my lowly camera :) look much nicer with lower levels of noise in Aperture than I have had in the past.

Previously I had used Topaz Denoise for 800 or above but now I only use it occasionally.

David

Brian Rybolt said...

What is meant by the phrase, "GEOMETRICALLY mended"?

I have been using Lightroom since it first appeared and can wholeheartedly recommend an upgrade to Lightroom 4. There have been major changes in the architecture plus the long awaited "soft proofing" feature for those of us who print. The noise reduction element of the application rivals all compeditors and better than most.

Brian

kirk tuck said...

The distortion caused by lens design is corrected. ie: barrel distortion, pin cushion distortion and more complex, non-linear distortions. Geometric distortions.

Condor said...

The basic problem here is noise at ISO400. This noise isn't Sony's fault, it's your fault. That's what the sensor is actually generating. This is because 24mp on a DX-size sensor is really a lot. That's about equivalent to 58mp on full frame. So, you are really pushing it here. "Noise reduction" is like pushing film. You're making the equipment do something it was not designed to do, and trying to deal with the consequences of that. Just shoot at ISO100, or whatever the lowest is, get good results, and be happy. If you want to shoot high ISO, like ISO1600, I'd move toward downrezing the files to 12mp or even 6mp. Or, just shoot on a Canon 5D, 12mp full frame.

Condor said...

Another analogy is with the people who, for generations, tried to squeeze every possible bit of performance out of a low-performance medium, 35mm film. Instead, a better solution is often to just use a bigger format. Use the gear in the middle of its comfort zone, not out at the edges.

Frank Grygier said...

Pushing any technology to it's limits is a natural behavior. Wringing the last bit of performance out of a computer, a camera, a car,yourself is the part of the passion. Comfort zones are no fun at all.

Jan Klier said...

Lightroom has become versatile enough that unless you need to handle composites, retouch at the pixel level, or do advanced color manipulation, it is certainly sufficient. Especially with LR4 which allows curves adjustments at the color channel level.

I would say that LR4 is good enough for 95% of photographers out there, including working pros. The only ones really needing Photoshop are those which work in advertising, architecture, fashion, or other 'high-touch' genres.

DGM said...

Thanks Jan,
I have been slowly coming to that conclusion. I never use the ultra advanced tools in Photoshop, but I have been a user since version 5. I think that came out back in the Jurassic period..... The last few upgrades were just for the RAW converter.

Thanks again.

dolan said...

Just one more person who is curious to hear what you think of LR4. It would be especially cool if you did a few LR3 vs LR4 raw conversions. Personally I found the upgrade well worth it; probably the best value "camera equipment" purchase this year.

kirk tuck said...

Condor, I take what you are saying with a massive grain of salt since the Nikon D800 uses a chip with a densely populated sensor and it currently yields better high ISO noise performance (according to DXO) than the well reviewed Canon 5Dmk2 and many other cameras with bigger pixel wells. It is also currently a dynamic range superstar.

My take is still that a process can be optimized for speed or quality at a price. As Jan said (paraphrasing) the total area of the triangle is inflexible.

I'm confident that I've improved the performance I get from the cameras by over a stop just by using appropriate software.

Bold Photography said...

I got stuck on shooting RAW a few years ago and almost never shoot JPG, unless I am shooting sports or somehow think that the camera can do better than I can in post (daylight @ iso100 is a good start) ... RAW does give me more latitude for being lazy in the shot, but that's no excuse for not getting it right in camera....

Gregg Mack said...

Kirk, I find it interesting that the reasons that you liek DXO Optics Pro aare some of the same reasons that I like Lightroom so much. ("the program will correct for the most common lens distortions, vignetting, chromatic aberrations, known sensor anomalies and other issues all at the same time. I can override the noise recipes in the program and fine tune each file if I like.")

Noise reduction was a very big improvement with LR3 and its Process Version 2010 (PV2010), and while LR4 uses a new PV2012 set of controls and algorithms, I have not seen anything that says that LR4 improved on the Noise Reduction of LR3.


Martin Evening gives a nice little demo on how the new PV2012 sliders are an improvement over the PV2010 sliders. I saw it today on DPReview here:

http://www.dpreview.com/articles/1205103502/extreme-contrast-edits-in-lightroom-4-and-acr-7

The best demonstration and explanation that I have seen on this topic was by Matt Kloskowski in this month's issue of Photoshop User magazine (for NAPP members), but that is not available as a PDF or any link that I can point you to. If you know of a NAPP member, maybe you could ask to see that article.

Carlo Santin said...

I shoot Raw pretty much exclusively and wouldn't ever consider shooting jpeg only. Olympus jpegs (from the digital Pens at least) are the only ones that look really good to my eyes, but even then I can still do much better working with the Olympus Raw. I use Adobe Camera Raw 6.6 and it is terrific, it has lens profiles for most lenses on the market. Then I open that into Photoshop CS5 if needed. I actually enjoy post processing and see it as an essential part of the photo, every bit as important as what happens when you press the shutter and before. This is where I am also ruthless with my own images in selecting only the ones that I think merit post production. If I shoot say 100 images, I might only pick the 5-10 of the best ones from that day and work them, maybe even fewer than that. I keep all of them, but they go into storage. The amount of time spent post processing quickly adds up, so I'm sure to only spend it on images that I deem worthy...I am an amateur of course so I don't have clients wishes to take into consideration, so no need for me to process 50 images if I don't want to.

Tony's Vision said...

As a long-time Photoshop user, and now a user of LR 4, I also must support the latter as useful for nearly all of my workflow. Once in a while I will pull an image from it into CS5 in order to combine different exposures rather than do the HDR thing. Or for switching heads around in group portraits, that sort of thing. But recently if I take an image out of LR it is more often into ColorEfex Pro 4.

Tony's Vision said...

Oh wait, then there is Photomerge and Content Aware Fill - so I guess my CS5 gets used more than I thought. But (getting back to the topic) LR4 does have a fine set of noise control tools.

Fletch said...

I've used various Sony bodies and don't really like the jpeg output. From my experience the noise reduction is too aggressive and it reduces the sharpness. With the FF Sony Alpha a900 I do everything I can to shoot at the lowest ISO possible, I shoot exclusively in RAW and then use Dfine from the Nik software suite to reduce noise on pretty much everything I do whereas with Fuji's x100 I'm happy to shoot up to ISO4000 and not worry too much.

Using the Dfine plugin for almost everything I do sounds quite involved (I'm probably that guy that spends too much butt time to the detriment of my images) but it's actually not that labour intensive. From Light Room: Photo > Edit in > Dfine 2.0> *opens up and analyses* > then click save. I'm happy with the automated result generated from the software, It's significantly better than anything I can achieve with lightroom sliders.

Condor said...

That may be true about the sensor of the D800, but the fact of the matter is, the sensor of the A77 and other Sony cameras with the same sensor (see Reichman's review of the A65 showing noise at ISO100 and a lot of noise at ISO400) generate a lot of noise. That is the characteristic of the hardware. It's just like Kodachrome. Looks great. Lots of detail. And it's ISO25. You can push it, if you want. You can even scan it and run the digital file through the noise filter of your choice. My point is, the noise is there. Personally, I would rather just shoot it where it's comfortable than kludge around with software fixes. If you want to do handheld in low light, use an X100.

kirk tuck said...

See the next post about the purchase of an a57. The polar opposite on the Sony noise spectrum...