There is a quite strong and (on face value) convincing argument for abandoning the scientific/lab analysis of the performance of lenses as stand-alone units or in combination with film or sensor performance. The argument goes like this: when a lens is good enough for the tasks that a photographer wants to accomplish and when this lens suits the style of the photographer there is no reason to check the absolute performance of the lens. The inherent image quality of the lens is made relative to the demands and views of the photographer. The yardstick or benchmark for the image quality is defined relative to the personal requirements of the photographer.
What, you may wish to ask, is the value of someone’s opinion, even if it comes from a highly esteemed or successful photographer, about a camera or a lens? If HCB (to give some example) would note that some lens he uses, let us say the Summarit 1.5/50mm, suits his photographic style and he is very happy with this lens, what can we conclude about the true or real-life performance of this lens?
It is true that the current practice of comparative testing (using test charts and some analysis software or using an elaborate three-dimensional setup with many solid objects) has been over-hyped and has lost much of the relevance it potentially might have. It makes no sense to note that lens A has 2123 lp/ph at the edge and at aperture x and lens B at the same position and aperture records 2187 lp/ph. It also makes no sense when one notes during a pixel-peeping session that there is a slight amount of color fringing at some unsharpness edge in the background and concludes that the lens in question is now less good than it could be or as might be expected by the reviewer. In this case the yardstick is an unrealistic ideal of optical performance. What we have now is rather depressing when we are interested in the true performance of a lens: counting line-pairs does not work, personal views are too individual and subjective, pixel-peeping is a dead-end. The practical photographer may repeat the remark that one is not interested in an assessment of factual image quality as long as the lens suits the personal style and requirements of the photographer. But imagine that we have two photographers one who asserts that the lens suits him well and another one who claims that this same lens is for him disappointing.
The only solution would be to establish the true performance of the lens. We have to make one important addition to this sentence and that is hat we want to establish the true performance that is relevant for the photographic practice in general. There are several aspects to consider: the image quality relative to the range of apertures, the image quality relative to the magnification or object distances (close, medium, infinity) and the image quality relative to the film or sensor area (center, corner, edge). The image quality itself might be defined as the definition of fine detail (contrast at the edges of textural detail) and the loss of contrast due to flare and inherent image aberrations. No lens is perfect and every designer will make a reasoned balance between the many conflicting demands. Any lens review or test should explain or demonstrate these choices and the effect they have on the inherent image quality and present the facts in such a way that the user can make an informed decision based on these facts.

Camera tests in the past were based on the view that cameras were precision-engineered mechanical products that could be dissected in several neatly separated parts and every part could be valued and measured according to some standard. Shutter speeds were measured, optical performance analyzed and film emulsions were compared in several developers. In many situations the necessary equipment was not available and reviewers tended to sketch personal opinions and practical experiences as a substitute.
Modern digital cameras however can no longer be reviewed in these technical-scientist and opiniated-experience traditions.
A modern camera (we can safely forget about the additional word ‘digital’ as every camera is a digital one) is no longer primarily a precision-engineered mechanical product, but an opto-mechatronic product. I wrote long ago about the transition from opto-mechanics to mechatronics. This does not imply that the precision is now less than in the past. On the contrary: mechatronics often demand a much higher level of precision than classical tools do, but mainly it is the difference that counts.
Take as example the recent Leica Q. This is a complex and integrated opto-mechatronic product that can hardly be reviewed in the classical traditions.The Q has a very slim body and a relatively large lens unit. This lens unit incorporates a leaf shutter with certain dimensions and the optical construction must be such that all the light collected at the front will pass through to the sensor. There is also an optical stabilizer that requires an additional lens group. The AF has to be fast and this requires a lens group with hardly a moving mass and also a stepper motor that must be located somewhere in the lens unit. Looking at the lens with only the specifics of aperture and focal length (as we do when looking at Leica M lenses) is the wrong way. The lens is now an opto-mechatronic unit and should be evaluated as such. This does not mean that classical parameters as performance should be neglected. Again on the contrary. The fact that the Q offers a digital zoom with the classical Tri-Elmar steps (28-35-50) implies that to hold the image quality over this range the performance of the lens must have a reserve or excess-performance that may not be visible when looking at the 28-setting, but in comparison with the 50-setting becomes important.
Modern and progressive automotive (car) journalists (not the types of Top Gear!) have made the transition to the view that a car must be viewed as a total and integrated concept that should fit in with current mobile lifestyles. The time that one could characterize a car by summing up a list of parameters (top speed, weight, fuel consumption and so on) is definitely gone as is the time that a review should be filled with personal statements (‘I like….’, ‘in my view….’ and so on).
The Q and even the T and X should be approached as a total package that supports or fits in or even enlarges the creative scope and concepts of the image maker/user, not as one of many tools for the almost obsolete concepts of street photography or landscape photography.

These might be simply letters in the alphabet or they may refer to cameras models marketed by a famous company in Wetzlar. There is hardly a year between the introduction of the renewed X ( with fixed 1.7/23 mm lens and APS-C CMOS sized sensor ) and the new Q (with fixed 1.7/28 mm lens and 135 CMOS sized sensor). In between the company also introduced the T with interchangeable lenses and also an APS-C CMOS sized sensor). All three seem to occupy the same area. Leica is increasingly profiling itself as the premium photographic company that combines a luxury brand with a technology brand. By the way: this is the same profile as that other iconic company is promoting: Apple. The Q (derived from the master engineer in the James Bond novels or from the Audi Q range) has a number of smart features and indeed the AF is really fast (at least in the prototype that I have been handling). Leica learned the lesson from the less than successful T that has a slower AF system. As I see it, Leica is at the crossroads. The M is the only product now in the Leica range without AF and they are under pressure to deliver a new system that would combine the iconic features of the M with AF. New lenses would be required and we see how problematic that is when one looks at the pace of introduction of T and S lenses. The main question is what would be the advantage of incorporating AF in the M system. The obvious answer is not the critical one. AF will certainly speed up the process of taking sharp pictures and many users of the M are old enough to have eye-sight problems. An M with AF lenses and EVF would hardly be different from any camera on the market nowadays. The T and Q systems have two functions: first to show that Leica is present in the lucrative high-end aficionado domain of buyers who refuse to accept that any current smartphone will produce comparable pictures. The second function is to demarcate the domain of the classical M. Every new introduction of an indigenously designed and manufactured Leica camera and a rebranded Panasonic camera poses a threat to the M system, because Leica lacks a clear marketing strategy and promotes every new camera with the same characteristics that relate to the classical RF values with some currently popular features thrown in. The Wired review of the Q notes that the Q has some advantages but with a price tag this high the company has a challenge to convince most prospective buyers to buy one. The reviews I have read are really trying to do just this. The Q is a most pleasurable camera to use and produces excellent quality results, but to say otherwise would be a disaster: when using a 24 Mp 24x36mm sized sensor and an excellent lens and current image-processing hard- and software one would not expect anything below premium quality. This is no longer the defining characteristic. Sad but true. Handling-wise the Q is a mix from elements that the X and T already offer and here we see a slow progress to one common specific Leica standard.
It has been noted (already long ago) that Leica is basically not a very innovative company (with the exception of the original Barnack camera) and is at its best when their engineers can improve on the results designed by others (see the M3). The original Leicaflex is a fine example of this approach and the Q fits in this tradition and seems to be a dedicated improvement on the Sony RX1.
As with the Leica T, the Leica Q is in danger of missing the boat. In the past there was a niche of high-tech high quality film-loading camera with advanced specifications (the Minolta TC-1 and Nikon 28ti are the obvious examples), but they did not succeed in the market. The Leica Q is very expensive and while its feature range is quite convincing, it does not justify this price tag.