Old and new lens MTF tests

By: markk 2824 days ago
Hi folks here
Photodo has been with me for years proving a very useful link for my past years in photography and I find the old MTF socre using 0-5 very practical and very correct to my own experience with Minolta, Nikon and Canon lens sharpness. However the newer version image test, though very comprehensive, has changed everything. Is there and editor here can explain the differnce? I have come across almost all newer tests and reviews posted but still feel confused. An example is Canon 100-400 IS I bought this week. According to old Photodo it scored 3.6, which is an acceptable mark but the newer test showed it to be a very bad one. Which should I trust?
By: inventory 2824 days ago
I know the lens Canon provided as a sample for that review was not as good as previous copies Ian has tested before.

All lenses will vary slightly, even if they are the same model.

An article on this occurence that may help you to understand a bit better can be found here

Ian tested three different Canon 100-400mm L IS lenses and they all came out different.

Some models, and some manufacturers have more of an issue with this variation than others. Hopefully as people start to build up the user reviews on this site, any lenses that have variable performance issues can be identified.

I'll look forward to seeing your verdict on the lens you bought!

Hope this helps.
By: IanA 2824 days ago
In the case you have highlighted, the lens supplied by Canon for the new test was a very old and battered one. The old Photodo score is probably around the right mark for a good one, however, beware of second hand ones being offered. They do suffer over time.

HIH

Ian
By: markk 2824 days ago
Inventory and IanA
Thanks for the reply. Perhaps 100-400 IS is NOT a good example because you have stated clearly in your review and users in their feedback for various different impression. However, how about say, the legendary 85/1.2 II. Older version 85/1.2 I scored very high I find this to be very true on my 30D. My another 85/1.4 G from Minolta does not perform quite close till stopping down to 2.5. Looking about on the old Photodo score, each lens was rated at 4.6 and 3.9 respectively, a very close representation of what I have. However, the newer 85/1.2 II test really confuses me. How should I interpret the newer MTF chart? Is it a bad lens without looking at your comment?

Thanks for reply
By: IanA 2823 days ago
The old tests were done on some very expensive equipment owned by Hassleblad, whereas the new tests are something you could actually do yourself if you wanted to put the kit together.

The problem with this is that the lens has to be mounted on a camera and a shot taken with it. That shot is then converted, as it has to be taken in RAW, to bypass the camera's optimisation settings. The resultant maximum quality Jpeg is then run through software called Imatest Pro and a number of parameters are measured and recorded.
The numbers from around 50 shots (f/2.8 zoom lens that stops down to f/32) are then fed into Excel in order to produce the graph.

What that graph tells you about every lens is where the lens performs best through the aperture range (with a prime lens) and at which focal lengths (with a zoom) you will get the best images.

The graph is a measure of the resolution achieved with the lens on whatever camera body it was tested on. As you know, IMAGE quality is not simply a result of resolution, but a combination of other factors also. Some of these factors are measurable and quantifiable, such as Chromatic aberation, but to produce a graph of it would mean some confusion, as the lowest figures would be the best, an inversion of the existing graph where the higher figures are best.
In order to covey that, I tell you in the text when it is good or bad. The same goes for contrast, which can affect image quality even more than resolution. It is not so easy to quantify and would add further confusion. The text, and sometimes the positive/negative bullets will tell you.

The other problem is sensors. They all vary, and there is no way I can test every lens on the same sensor. Therefore, you cannot compare the graphs directly against one another. This is true even of the same manufacturer. For example, I started off using a Canon 1D for Canon fit lenses, with a 4mp CCD sensor. Then along came EF-S lenses, which do not fit on the 1D, so I bought a 300D with it's 6mp CMOS sensor. When that gave up the ghost, I replaced it with a 20D and an 8mp CMOS sensor.

With each change of sensor comes a change in the 'standard' figures, wether I use cycles per pixel (cyl/pxl as on this site) or Line Widths per Picture Height (LW/PH) as in some other places.

There is also differences in other manufacturers sensors and also in the minimum ISO that the cameras are capable of.

What the graphs will tell you at a glance is:

Where the lens performs best
How good the edges are compared to the centre
How good one end of a zoom is compared to the other
How consistant the lens is through the aperture range.

I.E.

The lines reach a peak at f/5.6-8 means that is where the lens is performing at it's maximum resolution. If the lens is, for example, a Macro and it peaks at f/4, then it won't make a great macro lens but might do well in portraiture. If it peaks later, say f/8-11, it makes a better Macro as that is where it would be used more.

If the lines are close together, sometimes even crossing, then the edges are performing well compared to the centre.

The graphs are done at three focal lengths for zoom lenses, and these graphs can be compared to one another. Most zooms (although not all) are better at the shorter focal length and you can see from comparing, how much of a falloff in quality occurs as the focal length increases.

And finally, flat lines are good. If you get two lines, close together and fairly flat across the aperture range, you have the basis of a pretty good lens that is consistent from edge to edge and throughout the apertures.
Lenses with lumps in, and gaps between the lines, are only really good at their peaks (Although, if the peaks are in the right place they may still be good lenses. See above)

So, you can see that the new charts, although not directly comparable with the older scoring system, actually gives more information than it.

The 85mm f/1.2 II that you have asked about has, in the text, the answer to your question. It is a cracking one that is almost faultless and the only things I could mark it down on were a slightly quirky hood and the price.

I hope this helps you understand the new system.

Regards

Ian
By: markk 2823 days ago
Ian
Thank you for such detailed information. I think again Photodo will be my invaluable onsite information about lenses.
By: IanA 2823 days ago
You are welcome Mark.

Ian
By: janolofh 2819 days ago
At this page the information is wrong. MTF is for another lens:
http://www.photodo.com/product_452_p4.html
By: inventory 2818 days ago
The information has no been replaced with the correct test data.

If you spot any more errors on the site, please use the 'Suggest missing data' links that you'll find in the specification.

This way the Photodo team get an email notifying us of the problem so we can quickly apply any corrections needed.
By: mikecol 2818 days ago
Could you please comment (maybe update the "how we test lenses" link) on the RAW converter used and the settings (I noticed you used "no sharpening").

I understand you are using the JPEGs from the converter to feed into Imatest.

Thanks for the valuable information.

Mike
By: IanA 2818 days ago
RSE 2006. (Rawshooter Essentials) has been used for all the conversions on the tests to date (RSE 2005 prior to April).

The 'No Sharpening' setting is used in order to get the best possible result for the lens as opposed to the combination of camera/lens where each camera has differing sensors and sharpening has differing effects on the image. Imatest is capable of measuring all of the parameters for the system, whereas we are just trying to test the vagiaries of the lens.

With the early releases of Imatest, the only option was to use Jpegs and I have stuck with it both for consistancy and storage space (which becomes significant once you store an average 40 shots per lens in RAW plus Jpeg plus 3 .png files per shot with the results!)

I hope this helps.

Ian
By: longlens 2816 days ago
I was unable to locate any link on your excellent new website to the 'old' Photodo complete data. It listed many lenses no longer manufactured but still available used, and I found it invaluable in selecting lenses. I realize the data is old but it IS very accurate and many of us would appreciate having it also available in addition to the new information. Posting the 'Original Photodo Data' as a prominent link somewhere would be helpful. Thanks.
Stan
By: Will 2816 days ago
Stan, all the information from the old website has been moved accross. If you find any information is missing when you use our Lens search facility, please let us know.
By: petrox 2790 days ago
Hi Photodo, I am trying to find the old listing of the lens tests. Contax Carl Zeiss. The old website showed them all the new website does not show the 35-70 Zeiss vario zoom among others. Could you please show the old lens data as it was before please. Regards Peter
By: IanA 2790 days ago
Hi Peter,

The old version of Photodo is still here completely untouched.
I Hope it helps,

Ian
By: contaxfan 2306 days ago
Nikkor AF-S 300mm/4 test queston to IanA
Hi Ian

I suspect that you just took test data from older Nikkor AF 300mm/4 on your old site and put it for new version Nikkor AF-S 300mm/4 ED-IF lens, otherwise how to explain identical data for these two lenses which have different optics and different focusing distanse.
There is no any data for new Nikkor AF-S 300/4 lens on your old site, just for older Nikkor AF 300mm/4 lens. You started to test new lenses on your new site differently, showing only MTF graphs, only the Nikkor AF-S 300mm/4 shows exactly the same results as old AF 300/f on old site. I suspect, that these data are not from testing new AF-S 300mm/4, but were copied from older version Nikkor
AF 300/4, what is confusing me. Nikkor AF-S 300/4 is my favorite lens and I 'd like to know the difference in your tests between these two Nikkors.
Hope to hear your answer
Thank you
Val, contaxfan
By: EricD 2214 days ago
I was confused by the change from the familiar 'contrast vs spatial frequency' (% vs lp/mm) to 'MTF50 value in cyl/pxl vs aperture'.

I would assume that is the spatial frequency at which contrast is halved (50%), expressed as the ratio of spatial frequency (cycles) to pixel-size ?

Presumably we also need to know that a Nikon D50 has its pixels on a 6 micro-metre pitch and adjust to match our own pixel size ? I'm not sure that adjusting is simple. Half the pixel-pitch probably won't mean half the contrast.

What is the pixel-pitch of a D70 ?

Is 0.5 the theoretical maximum possible for 'MTF50 value in cyl/pxl' ? I'm uncomfortable with the mixture of analogue and digital - over how many pixels is the response measured ? What happens when the response wiggles up & down a bit, so that it passes through 50% several times ? (ie Aliasing)

Another couple of factors are the degree of smoothing applied in-camera : both by the optical anti-aliasing filter in front of the sensor (rough glass), and the smoothing filter applied in software. Some Nikon cameras' RAW files are not really raw - there is considerable processing, aiming to reduce moire/aliasing when periodic objects in the image approach pixel-sizes. ie so that points of white light don't just fall on green sensors and appear green.
http://www.astrosurf.org/buil/d70v10d/eval.htm

Also, when you test a macro/micro lens, are you testing at 1:1 or normal shooting distances ?

The camera has such a large effect that it is better to test MTF of a lens on its own, and the MTF of the camera separately. Then the two can be combined to reflect reality.

If we happen to have the same camera, the test will reflect reality - but a 3rd-party (eg Sigma) lens may perform quite differently on a Canon and on a Nikon or on a 'consumer 6MP' or a 'pro 12MP' camera !

Someone said 'flat lines are good' - but low flat lines are not as good as peaky higher lines ! Who cares if the response is the same at all apertures ? Just use the best aperture, if you can. Given that small apertures will always be diffraction limited, flat lines are not good, in my opinion.
By: IanA 2214 days ago
Flat lines are good in that it shows a consistant optic. Tailing off due to diffraction is inevitable.

"Also, when you test a macro/micro lens, are you testing at 1:1 or normal shooting distances "

Tests are carried out at 25x the focal length.

Ian
By: helloangel 36 days ago
Spam removed.

Add your message

Login required
Please login here or if you've not registered, you can register here. Registering is safe, quick and free.