Tight crop from Panasonic fz2500
The Original uncropped frame
I'm not a big fan of gobbledy-gook, jargon and half-understood scientific word constructs meant to justify a visceral opinion in the service of marketing (and don't get me started on the satanic nature of acronyms). By this I mean that having a rationale for why something should be better or worse is not the same as a camera or lens actually being better or worse. So much of testing is still very subjective and, when it comes to issues such as focus, current cameras have far too much complexity for most users, which seems to exponentially (see what I did just there?) increase the things that can go wrong; or be mis-set.
Two recent things affected my ability to believe without question the results of one of the most famous camera review sites on the web. The first was their declaration that the Leica lens on the front of the Panasonic fz 2500 was mediocre. I was able to prove (at least to myself) that much, if not all, of the softness some people were experiencing with that lens had to do with the automatic focusing modes and their interface with the touchscreen, and the tenuous software that binds them together. If the camera is set up correctly for your individual use targets it is capable of lens quality performance rivaling its closest rivals.
Some tinkering with focus modes should have given the wayward reviewers more insight, at least into the quality of the lens itself, so they could re-focus their attention to the vagaries (not faults?) of the focusing system itself. The bottom line is that the Panasonic bridge camera is capable of making wonderfully sharp images, in the right hands.
But the final, jarring, sledgehammer blow to the credibility of this corporate band of reviewers has been the ongoing exuberant praise, and alternate active rehabilitation, of the Sony a9 camera. A camera which sets the record for the most lines of text written in the service of naked marketing ever seen in the hyperbolic history of camera reviewing.
The coup de grace to the credibility of the site in question was their re-re-testing of the a9's sharpness via a series of tests, the methods of which diverged from the parameters of tests done with hundreds of other cameras, for no other reason than to increase the sharpness score for that particular camera. Of course, a new testing procedure means that none of the previously tested cameras can be objectively compared, on that site, with the a9 because they were not given the endless chances to finally excel which have been lavished on the Sony product. Nor were test procedures previously modified to compensate for the shortcomings of other products. If you want objectivity and also want to believe in the scientific method you can't have it both ways.
Just jotting down "fibonacci sequence" doesn't validate method. (They never mentioned Fibannaci Sequence but I'm making a point about trying to intimidate readers by trotting out phrases or arcane procedures that just don't match the situation...).
I sympathize with the review site. It's a tough way to make a living in the post camera buying era. Click throughs become absolutely critical. But I find there's no substitute to living with a camera for a sustained period of time in order to understand it on a more holistic, even visceral, level. Most of the current cameras can only be assessed as part of a system. I prefer "hands-on" shooting to chart tests. This is not "String Theory" and the reviewers are not all Phd. researchers at Cal Tech.
Just to be clear: Objective testing should mean all cameras get tested the same way.
Now, if the reviewers want some non-Sony a9 work that would actually be continuously helpful to real photographers, who want to know if they should buy a certain piece of gear, they should consider re-reviewing cameras that have already been reviewed each time a big firmware fix is unveiled. There is much consensus that some cameras have been made amazingly better by new firmware and yet the old reviews stand as fact. The world iterates. Reviews should too. Right up until the camera in question is retired from the market.
(no ad for the Sony a9 here...).
added 6-27: An interesting article by Erwin Puts about testing and manufacture tolerances: http://www.imx.nl/photo/optics/optics/page62.html