Smokey
11-05-2013, 08:15 PM
Jim Willcox who is Senior Editor/Electronics at CR recently let one of AVSForum member take a tour of CR lab facilities at Yonkers, NY. It is an interesting read and show vigoruos their testing is.
From AVSForum (full link with more info at the bottom):
http://www.avsforum.com/content/type/61/id/297708/width/500/height/1000/flags/LL
Consumer Reports National Testing and Research Center in Yonkers, NY
Over the course of one year, Consumer Reports tests and reviews over 200 different TV models. Every unit goes through exactly the same battery of tests, all designed to reveal the full capabilities of each television. Nothing is taken for granted—even something as basic as whether a TV is 720p or 1080p is subject to independent verification—because printed specifications are not always accurate.
http://www.avsforum.com/content/type/61/id/297709/width/500/height/1000/flags/LL
The lab is currently set up to test thirty TVs at one time.
Claudio Ciacci is responsible for how Consumer Reports tests televisions today. It takes five weeks for a TV to pass through the lab, and the lab tests 30 TVs in that time period. Technicians subject each unit to a series of tests that measure its capabilities while simultaneously identifying the optimum settings for that display.
For example, when it comes to discount televisions, manufacturers sometimes use panels from different suppliers in different-sized televisions within the same line. As a result, performance is not consistent throughout all screen sizes, which is why it is necessary to test all the different screen sizes. Another factor is firmware version, because sometimes a firmware update can fundamentally change the settings that are optimal for a given television.
The point of the CR lab's process is to get the televisions to the best possible state in terms of picture quality, short of a professional calibration. As Claudio explained it, if a television really does require professional calibration to look great, that will actually count against it. The reason is simple—many modern televisions actually do perform very well out of the box. Of course, it also counts against a TV if standard tweaking does not result in a good picture.
By design, the tests are as objective as possible; any trained technician can take the place of any other trained technician and produce essentially the same results. The technicians log the optimum settings, along with the accompanying firmware version. At this point, the TV moves on to the final testing phase.
http://www.avsforum.com/content/type/61/id/297723/width/500/height/1000/flags/LL
TVs lined up for testing
The testing and review process is dauntingly technical. I saw the workbook in which all the collected data resides—it is amazing that Consumer Reports can distill so much information into its traditional method of ranking from poor to excellent. When one considers the explosion of features that are the hallmark of modern TVs, it becomes clear what a Herculean task it is to stay on top of it all.
The first thing the lab does is measure basic performance parameters—contrast, sharpness, color, resolution, native refresh rate, power consumption, etc.—as well as compile a list of features. The technicians go over each unit with a fine-tooth comb and essentially create a new spec sheet from scratch. Every part of the testing process is proprietary to Consumer Reports, including the test patterns.
http://www.avsforum.com/content/type/61/id/297711/width/500/height/1000/flags/LL
A Photo Research SpectraScan PR-740 spectroradiometer takes hyper-accurate measurements
If a TV has premium features such as 3D capability, those also are tested. One of the more impressive examples revealed the amount of 3D ghosting generated by a given TV. I was bit shocked by the sorry state of active-shutter 3D technology, especially when it comes to LCD panels. For example, I did not know that crosstalk was such a big issue—thanks to Claudio's tests, its ubiquitous presence on active-shutter panels was obvious.
http://www.avsforum.com/content/type/61/id/297713/width/500/height/1000/flags/LL
Claudio Ciacci explaining the testing process
Once I was in the lab, I began to grasp how serious Consumer Reports is about its TV-testing program. As Jim and Claudio explained it, in years past, the testing process took too long to be relevant, considering the frantic pace of development in the flat-panel arena. With traditional print publishing, it could take up to six months for a new TV model's review to appear—and even then, the sheer number of makes, models, and screen sizes made researching a TV purchase a daunting process. Fortunately, websites bypass all of these limitations, allowing for timely publication of reviews while incorporating search and sort tools that make comparisons fast and easy. This is crucial, because the pace of change in the TV industry is literally breakneck—it truly is survival of the fittest.
I asked Claudio about his opinions regarding video processing, such as noise reduction, so-called "vivid mode," and frame interpolation. He is totally opposed to gratuitous video processing, which came as a great relief. Part of the CR testing process includes confirming that the video-processing features—which are normally on by default—can be disabled by the user. This is not always the case with all TVs, especially when it comes to budget brands. Turning a feature off does not always completely disable processing.
http://www.avsforum.com/content/type/61/id/297716/width/500/height/1000/flags/LL
Some of the test patterns and scenes used by Consumer Reports lab
One of the more punishing tests for edgelit LCD TVs is watching a simple white object traveling horizontally across a black background. Manufacturers often claim some form of "local dimming" for these units, but the simple fact is there's no way to isolate the light coming from the edges to illuminate a bright object in the center of the screen. An artifact known as flashlighting is clearly visible, as is an amorphous bright blob around the white object—a halo artifact.
In the same room, a calibrated high-end plasma television served as a reference unit. The white object tracked perfectly on the plasma, with no halo or flashlight artifact. The combined difference in black levels and lack of artifacts demonstrated a stunning gap in performance between the two technologies. Plasma fans certainly know what they are talking about when it comes to image quality issues and edgelit LCD.
http://www.avsforum.com/content/type/61/id/297717/width/500/height/1000/flags/LL
Edge lit LCD TVs suffer from halo artifacts when rendering bright objects on dark backgrounds
There is a reference-quality plasma television in the lab—a fully calibrated Panasonic TC-P55VT50. Actually, there are two of them. Each television has to go up against this reference, which is a tough TV to beat. Then, the techs watch a variety of content on the optimized TV in order to get a handle on how it performs an actual use, as opposed to how it measures. Ultimately, both factors are relevant to any thorough review.
At the current pace, Consumer Reports will test 300 different units each year. I do not know of any other outlet that is testing that many televisions as thoroughly and scientifically. I want to thank Jim Willcox for making my visit possible, and Claudio Ciacci for taking the time to show me the lab. I also want to thank engineers Christopher Andrade and Matt Ferretti, who were present on the tour and do a lot of the heavy lifting in the lab.
http://forums.audioreview.com/editpost.php?do=updatepost&p=397625
From AVSForum (full link with more info at the bottom):
http://www.avsforum.com/content/type/61/id/297708/width/500/height/1000/flags/LL
Consumer Reports National Testing and Research Center in Yonkers, NY
Over the course of one year, Consumer Reports tests and reviews over 200 different TV models. Every unit goes through exactly the same battery of tests, all designed to reveal the full capabilities of each television. Nothing is taken for granted—even something as basic as whether a TV is 720p or 1080p is subject to independent verification—because printed specifications are not always accurate.
http://www.avsforum.com/content/type/61/id/297709/width/500/height/1000/flags/LL
The lab is currently set up to test thirty TVs at one time.
Claudio Ciacci is responsible for how Consumer Reports tests televisions today. It takes five weeks for a TV to pass through the lab, and the lab tests 30 TVs in that time period. Technicians subject each unit to a series of tests that measure its capabilities while simultaneously identifying the optimum settings for that display.
For example, when it comes to discount televisions, manufacturers sometimes use panels from different suppliers in different-sized televisions within the same line. As a result, performance is not consistent throughout all screen sizes, which is why it is necessary to test all the different screen sizes. Another factor is firmware version, because sometimes a firmware update can fundamentally change the settings that are optimal for a given television.
The point of the CR lab's process is to get the televisions to the best possible state in terms of picture quality, short of a professional calibration. As Claudio explained it, if a television really does require professional calibration to look great, that will actually count against it. The reason is simple—many modern televisions actually do perform very well out of the box. Of course, it also counts against a TV if standard tweaking does not result in a good picture.
By design, the tests are as objective as possible; any trained technician can take the place of any other trained technician and produce essentially the same results. The technicians log the optimum settings, along with the accompanying firmware version. At this point, the TV moves on to the final testing phase.
http://www.avsforum.com/content/type/61/id/297723/width/500/height/1000/flags/LL
TVs lined up for testing
The testing and review process is dauntingly technical. I saw the workbook in which all the collected data resides—it is amazing that Consumer Reports can distill so much information into its traditional method of ranking from poor to excellent. When one considers the explosion of features that are the hallmark of modern TVs, it becomes clear what a Herculean task it is to stay on top of it all.
The first thing the lab does is measure basic performance parameters—contrast, sharpness, color, resolution, native refresh rate, power consumption, etc.—as well as compile a list of features. The technicians go over each unit with a fine-tooth comb and essentially create a new spec sheet from scratch. Every part of the testing process is proprietary to Consumer Reports, including the test patterns.
http://www.avsforum.com/content/type/61/id/297711/width/500/height/1000/flags/LL
A Photo Research SpectraScan PR-740 spectroradiometer takes hyper-accurate measurements
If a TV has premium features such as 3D capability, those also are tested. One of the more impressive examples revealed the amount of 3D ghosting generated by a given TV. I was bit shocked by the sorry state of active-shutter 3D technology, especially when it comes to LCD panels. For example, I did not know that crosstalk was such a big issue—thanks to Claudio's tests, its ubiquitous presence on active-shutter panels was obvious.
http://www.avsforum.com/content/type/61/id/297713/width/500/height/1000/flags/LL
Claudio Ciacci explaining the testing process
Once I was in the lab, I began to grasp how serious Consumer Reports is about its TV-testing program. As Jim and Claudio explained it, in years past, the testing process took too long to be relevant, considering the frantic pace of development in the flat-panel arena. With traditional print publishing, it could take up to six months for a new TV model's review to appear—and even then, the sheer number of makes, models, and screen sizes made researching a TV purchase a daunting process. Fortunately, websites bypass all of these limitations, allowing for timely publication of reviews while incorporating search and sort tools that make comparisons fast and easy. This is crucial, because the pace of change in the TV industry is literally breakneck—it truly is survival of the fittest.
I asked Claudio about his opinions regarding video processing, such as noise reduction, so-called "vivid mode," and frame interpolation. He is totally opposed to gratuitous video processing, which came as a great relief. Part of the CR testing process includes confirming that the video-processing features—which are normally on by default—can be disabled by the user. This is not always the case with all TVs, especially when it comes to budget brands. Turning a feature off does not always completely disable processing.
http://www.avsforum.com/content/type/61/id/297716/width/500/height/1000/flags/LL
Some of the test patterns and scenes used by Consumer Reports lab
One of the more punishing tests for edgelit LCD TVs is watching a simple white object traveling horizontally across a black background. Manufacturers often claim some form of "local dimming" for these units, but the simple fact is there's no way to isolate the light coming from the edges to illuminate a bright object in the center of the screen. An artifact known as flashlighting is clearly visible, as is an amorphous bright blob around the white object—a halo artifact.
In the same room, a calibrated high-end plasma television served as a reference unit. The white object tracked perfectly on the plasma, with no halo or flashlight artifact. The combined difference in black levels and lack of artifacts demonstrated a stunning gap in performance between the two technologies. Plasma fans certainly know what they are talking about when it comes to image quality issues and edgelit LCD.
http://www.avsforum.com/content/type/61/id/297717/width/500/height/1000/flags/LL
Edge lit LCD TVs suffer from halo artifacts when rendering bright objects on dark backgrounds
There is a reference-quality plasma television in the lab—a fully calibrated Panasonic TC-P55VT50. Actually, there are two of them. Each television has to go up against this reference, which is a tough TV to beat. Then, the techs watch a variety of content on the optimized TV in order to get a handle on how it performs an actual use, as opposed to how it measures. Ultimately, both factors are relevant to any thorough review.
At the current pace, Consumer Reports will test 300 different units each year. I do not know of any other outlet that is testing that many televisions as thoroughly and scientifically. I want to thank Jim Willcox for making my visit possible, and Claudio Ciacci for taking the time to show me the lab. I also want to thank engineers Christopher Andrade and Matt Ferretti, who were present on the tour and do a lot of the heavy lifting in the lab.
http://forums.audioreview.com/editpost.php?do=updatepost&p=397625