Visiting ConsumerReports TV Testing Lab [Archive] - Audio & Video Forums


View Full Version : Visiting ConsumerReports TV Testing Lab

11-05-2013, 08:15 PM
Jim Willcox who is Senior Editor/Electronics at CR recently let one of AVSForum member take a tour of CR lab facilities at Yonkers, NY. It is an interesting read and show vigoruos their testing is.

From AVSForum (full link with more info at the bottom):
Consumer Reports National Testing and Research Center in Yonkers, NY

Over the course of one year, Consumer Reports tests and reviews over 200 different TV models. Every unit goes through exactly the same battery of tests, all designed to reveal the full capabilities of each television. Nothing is taken for granted—even something as basic as whether a TV is 720p or 1080p is subject to independent verification—because printed specifications are not always accurate.
The lab is currently set up to test thirty TVs at one time.

Claudio Ciacci is responsible for how Consumer Reports tests televisions today. It takes five weeks for a TV to pass through the lab, and the lab tests 30 TVs in that time period. Technicians subject each unit to a series of tests that measure its capabilities while simultaneously identifying the optimum settings for that display.

For example, when it comes to discount televisions, manufacturers sometimes use panels from different suppliers in different-sized televisions within the same line. As a result, performance is not consistent throughout all screen sizes, which is why it is necessary to test all the different screen sizes. Another factor is firmware version, because sometimes a firmware update can fundamentally change the settings that are optimal for a given television.

The point of the CR lab's process is to get the televisions to the best possible state in terms of picture quality, short of a professional calibration. As Claudio explained it, if a television really does require professional calibration to look great, that will actually count against it. The reason is simple—many modern televisions actually do perform very well out of the box. Of course, it also counts against a TV if standard tweaking does not result in a good picture.

By design, the tests are as objective as possible; any trained technician can take the place of any other trained technician and produce essentially the same results. The technicians log the optimum settings, along with the accompanying firmware version. At this point, the TV moves on to the final testing phase.
TVs lined up for testing

The testing and review process is dauntingly technical. I saw the workbook in which all the collected data resides—it is amazing that Consumer Reports can distill so much information into its traditional method of ranking from poor to excellent. When one considers the explosion of features that are the hallmark of modern TVs, it becomes clear what a Herculean task it is to stay on top of it all.

The first thing the lab does is measure basic performance parameters—contrast, sharpness, color, resolution, native refresh rate, power consumption, etc.—as well as compile a list of features. The technicians go over each unit with a fine-tooth comb and essentially create a new spec sheet from scratch. Every part of the testing process is proprietary to Consumer Reports, including the test patterns.
A Photo Research SpectraScan PR-740 spectroradiometer takes hyper-accurate measurements

If a TV has premium features such as 3D capability, those also are tested. One of the more impressive examples revealed the amount of 3D ghosting generated by a given TV. I was bit shocked by the sorry state of active-shutter 3D technology, especially when it comes to LCD panels. For example, I did not know that crosstalk was such a big issue—thanks to Claudio's tests, its ubiquitous presence on active-shutter panels was obvious.
Claudio Ciacci explaining the testing process

Once I was in the lab, I began to grasp how serious Consumer Reports is about its TV-testing program. As Jim and Claudio explained it, in years past, the testing process took too long to be relevant, considering the frantic pace of development in the flat-panel arena. With traditional print publishing, it could take up to six months for a new TV model's review to appear—and even then, the sheer number of makes, models, and screen sizes made researching a TV purchase a daunting process. Fortunately, websites bypass all of these limitations, allowing for timely publication of reviews while incorporating search and sort tools that make comparisons fast and easy. This is crucial, because the pace of change in the TV industry is literally breakneck—it truly is survival of the fittest.

I asked Claudio about his opinions regarding video processing, such as noise reduction, so-called "vivid mode," and frame interpolation. He is totally opposed to gratuitous video processing, which came as a great relief. Part of the CR testing process includes confirming that the video-processing features—which are normally on by default—can be disabled by the user. This is not always the case with all TVs, especially when it comes to budget brands. Turning a feature off does not always completely disable processing.
Some of the test patterns and scenes used by Consumer Reports lab

One of the more punishing tests for edgelit LCD TVs is watching a simple white object traveling horizontally across a black background. Manufacturers often claim some form of "local dimming" for these units, but the simple fact is there's no way to isolate the light coming from the edges to illuminate a bright object in the center of the screen. An artifact known as flashlighting is clearly visible, as is an amorphous bright blob around the white object—a halo artifact.

In the same room, a calibrated high-end plasma television served as a reference unit. The white object tracked perfectly on the plasma, with no halo or flashlight artifact. The combined difference in black levels and lack of artifacts demonstrated a stunning gap in performance between the two technologies. Plasma fans certainly know what they are talking about when it comes to image quality issues and edgelit LCD.
Edge lit LCD TVs suffer from halo artifacts when rendering bright objects on dark backgrounds

There is a reference-quality plasma television in the lab—a fully calibrated Panasonic TC-P55VT50. Actually, there are two of them. Each television has to go up against this reference, which is a tough TV to beat. Then, the techs watch a variety of content on the optimized TV in order to get a handle on how it performs an actual use, as opposed to how it measures. Ultimately, both factors are relevant to any thorough review.

At the current pace, Consumer Reports will test 300 different units each year. I do not know of any other outlet that is testing that many televisions as thoroughly and scientifically. I want to thank Jim Willcox for making my visit possible, and Claudio Ciacci for taking the time to show me the lab. I also want to thank engineers Christopher Andrade and Matt Ferretti, who were present on the tour and do a lot of the heavy lifting in the lab.

11-08-2013, 04:23 AM
I have been a subscriber to Consumer Reports since the 60's. I have to admit that their testing methods have improved by leaps and bounds over some of those once used in the past.

Having worked in the Consumer Electronics Industry, I always cringed whenever CR came out with a report on stereo receivers, phono cartridges or loudspeakers. Almost none of us in the business ever agreed with anything CR had to say, and particularly objected to their manner of testing loudspeakers, rating performance based on accuracy as measured in an anechoic chamber and nothing else. AR loudspeakers just about always topped the list, and when testing what CR called "expensive" loudspeakers, a Marantz model came in first, with an ESS model next, all way ahead of the Dahlquist DQ-10 - a considerably better speaker in the minds of anyone involved in the industry. I even worked for ESS at the time and thought the tests were nuts.

When testing breakfast cereals, CR fed laboratory rats cereal and water, then measured the rate of growth for the rats, claiming that the rats had similar nutritional needs as humans. The top-rated cereal was almost always Kellogg's Special K. On one of the first such tests, Shredded Wheat came in near the bottom of the ratings, but on a subsequent test - using the identical methods - Shredded Wheat came out near the top. CR had no explanation for the disparity.

Today, CR is a "whole nuthuh'" ball game, and a far more reliable source of information.

11-09-2013, 05:52 AM
My dad has been a long time subscriber to consumer reports and still is but often times it was a toss up if i agreed with their reviews or not. I always thought their write ups of automobiles was interesting.

AVSForum is awful.

harley .guy07
11-09-2013, 10:55 AM
My father swore by what CR said in their reports. Its funny looking at when I was younger and first getting into this hobby I would read CR and see that they had a complete love for Bose products and always gave them top ratings when they tested speakers. I surely did not agree with them on some of the Audio ratings but televisions and other electronics I could see real truth in their ratings. And their ratings for vehicles always interested me because they have for years gave the Japanese auto makers top ratings every year since the late 80's or early 90's and just recently they are starting to give the American car companies top ratings over some Japaneses companies now. It is just interesting how things change over time and how biases work and how they can change.

11-11-2013, 09:36 PM
I also been following CR since 80's, mainly their TV ratings. They have been pretty consistent in accuracy of their ratings, but ratings is not always 100% correct all the time. And most of it can be attributed to TV's own variations and performance differences within a model number.

AVSForum is awful.

Members can be get little bit cocky, but there also alot of good information and knowledgable poeple on that site...such as editor of CR Jim Wilcox :)

11-20-2013, 11:30 PM
It was a nice visit to TV Testing Lab.

rabi cs
12-11-2013, 09:58 PM
Also have a B&O System from the 1980s, a Carver System from the 1980s, and an 80s Revox system.