Results 1 to 7 of 7
  1. #1
    Suspended Smokey's Avatar
    Join Date
    Jan 2004

    Smile Visiting ConsumerReports TV Testing Lab

    Jim Willcox who is Senior Editor/Electronics at CR recently let one of AVSForum member take a tour of CR lab facilities at Yonkers, NY. It is an interesting read and show vigoruos their testing is.

    From AVSForum (full link with more info at the bottom):

    Consumer Reports National Testing and Research Center in Yonkers, NY

    Over the course of one year, Consumer Reports tests and reviews over 200 different TV models. Every unit goes through exactly the same battery of tests, all designed to reveal the full capabilities of each television. Nothing is taken for granted—even something as basic as whether a TV is 720p or 1080p is subject to independent verification—because printed specifications are not always accurate.

    The lab is currently set up to test thirty TVs at one time.

    Claudio Ciacci is responsible for how Consumer Reports tests televisions today. It takes five weeks for a TV to pass through the lab, and the lab tests 30 TVs in that time period. Technicians subject each unit to a series of tests that measure its capabilities while simultaneously identifying the optimum settings for that display.

    For example, when it comes to discount televisions, manufacturers sometimes use panels from different suppliers in different-sized televisions within the same line. As a result, performance is not consistent throughout all screen sizes, which is why it is necessary to test all the different screen sizes. Another factor is firmware version, because sometimes a firmware update can fundamentally change the settings that are optimal for a given television.

    The point of the CR lab's process is to get the televisions to the best possible state in terms of picture quality, short of a professional calibration. As Claudio explained it, if a television really does require professional calibration to look great, that will actually count against it. The reason is simple—many modern televisions actually do perform very well out of the box. Of course, it also counts against a TV if standard tweaking does not result in a good picture.

    By design, the tests are as objective as possible; any trained technician can take the place of any other trained technician and produce essentially the same results. The technicians log the optimum settings, along with the accompanying firmware version. At this point, the TV moves on to the final testing phase.

    TVs lined up for testing

    The testing and review process is dauntingly technical. I saw the workbook in which all the collected data resides—it is amazing that Consumer Reports can distill so much information into its traditional method of ranking from poor to excellent. When one considers the explosion of features that are the hallmark of modern TVs, it becomes clear what a Herculean task it is to stay on top of it all.

    The first thing the lab does is measure basic performance parameters—contrast, sharpness, color, resolution, native refresh rate, power consumption, etc.—as well as compile a list of features. The technicians go over each unit with a fine-tooth comb and essentially create a new spec sheet from scratch. Every part of the testing process is proprietary to Consumer Reports, including the test patterns.

    A Photo Research SpectraScan PR-740 spectroradiometer takes hyper-accurate measurements

    If a TV has premium features such as 3D capability, those also are tested. One of the more impressive examples revealed the amount of 3D ghosting generated by a given TV. I was bit shocked by the sorry state of active-shutter 3D technology, especially when it comes to LCD panels. For example, I did not know that crosstalk was such a big issue—thanks to Claudio's tests, its ubiquitous presence on active-shutter panels was obvious.

    Claudio Ciacci explaining the testing process

    Once I was in the lab, I began to grasp how serious Consumer Reports is about its TV-testing program. As Jim and Claudio explained it, in years past, the testing process took too long to be relevant, considering the frantic pace of development in the flat-panel arena. With traditional print publishing, it could take up to six months for a new TV model's review to appear—and even then, the sheer number of makes, models, and screen sizes made researching a TV purchase a daunting process. Fortunately, websites bypass all of these limitations, allowing for timely publication of reviews while incorporating search and sort tools that make comparisons fast and easy. This is crucial, because the pace of change in the TV industry is literally breakneck—it truly is survival of the fittest.

    I asked Claudio about his opinions regarding video processing, such as noise reduction, so-called "vivid mode," and frame interpolation. He is totally opposed to gratuitous video processing, which came as a great relief. Part of the CR testing process includes confirming that the video-processing features—which are normally on by default—can be disabled by the user. This is not always the case with all TVs, especially when it comes to budget brands. Turning a feature off does not always completely disable processing.

    Some of the test patterns and scenes used by Consumer Reports lab

    One of the more punishing tests for edgelit LCD TVs is watching a simple white object traveling horizontally across a black background. Manufacturers often claim some form of "local dimming" for these units, but the simple fact is there's no way to isolate the light coming from the edges to illuminate a bright object in the center of the screen. An artifact known as flashlighting is clearly visible, as is an amorphous bright blob around the white object—a halo artifact.

    In the same room, a calibrated high-end plasma television served as a reference unit. The white object tracked perfectly on the plasma, with no halo or flashlight artifact. The combined difference in black levels and lack of artifacts demonstrated a stunning gap in performance between the two technologies. Plasma fans certainly know what they are talking about when it comes to image quality issues and edgelit LCD.

    Edge lit LCD TVs suffer from halo artifacts when rendering bright objects on dark backgrounds

    There is a reference-quality plasma television in the lab—a fully calibrated Panasonic TC-P55VT50. Actually, there are two of them. Each television has to go up against this reference, which is a tough TV to beat. Then, the techs watch a variety of content on the optimized TV in order to get a handle on how it performs an actual use, as opposed to how it measures. Ultimately, both factors are relevant to any thorough review.

    At the current pace, Consumer Reports will test 300 different units each year. I do not know of any other outlet that is testing that many televisions as thoroughly and scientifically. I want to thank Jim Willcox for making my visit possible, and Claudio Ciacci for taking the time to show me the lab. I also want to thank engineers Christopher Andrade and Matt Ferretti, who were present on the tour and do a lot of the heavy lifting in the lab.
    Last edited by Smokey; 11-05-2013 at 09:10 PM.

  2. #2
    Aging Smartass
    Join Date
    Oct 2005
    Moore, SC
    I have been a subscriber to Consumer Reports since the 60's. I have to admit that their testing methods have improved by leaps and bounds over some of those once used in the past.

    Having worked in the Consumer Electronics Industry, I always cringed whenever CR came out with a report on stereo receivers, phono cartridges or loudspeakers. Almost none of us in the business ever agreed with anything CR had to say, and particularly objected to their manner of testing loudspeakers, rating performance based on accuracy as measured in an anechoic chamber and nothing else. AR loudspeakers just about always topped the list, and when testing what CR called "expensive" loudspeakers, a Marantz model came in first, with an ESS model next, all way ahead of the Dahlquist DQ-10 - a considerably better speaker in the minds of anyone involved in the industry. I even worked for ESS at the time and thought the tests were nuts.

    When testing breakfast cereals, CR fed laboratory rats cereal and water, then measured the rate of growth for the rats, claiming that the rats had similar nutritional needs as humans. The top-rated cereal was almost always Kellogg's Special K. On one of the first such tests, Shredded Wheat came in near the bottom of the ratings, but on a subsequent test - using the identical methods - Shredded Wheat came out near the top. CR had no explanation for the disparity.

    Today, CR is a "whole nuthuh'" ball game, and a far more reliable source of information.

  3. #3
    Charm Thai™
    Join Date
    Feb 2002
    Cleveland, Ohio
    My dad has been a long time subscriber to consumer reports and still is but often times it was a toss up if i agreed with their reviews or not. I always thought their write ups of automobiles was interesting.

    AVSForum is awful.

  4. #4
    Forum Regular harley .guy07's Avatar
    Join Date
    Aug 2009
    Springfield, Mo
    My father swore by what CR said in their reports. Its funny looking at when I was younger and first getting into this hobby I would read CR and see that they had a complete love for Bose products and always gave them top ratings when they tested speakers. I surely did not agree with them on some of the Audio ratings but televisions and other electronics I could see real truth in their ratings. And their ratings for vehicles always interested me because they have for years gave the Japanese auto makers top ratings every year since the late 80's or early 90's and just recently they are starting to give the American car companies top ratings over some Japaneses companies now. It is just interesting how things change over time and how biases work and how they can change.

    Marantz SR5008(HT)
    Nu Force P8 Preamp (2 channel)
    Pass Labs X150.5(2 channel)
    Adcom 545 mk2 power amp(rear channel amp)
    Spatial Audio M3 Turbo S Mains Speakers
    Dayton 8" HO custom sealed subwoofer(2 channel)
    Yamaha NS-c444 center channel
    Emotiva ERD-1 surround speakers
    JBL e250p subwoofer highly modified
    Samsung 46" LED TV
    OPPO BDP-83 blue ray/multi format player
    ps-audio NuWave dac (2 channel)
    Dell I660 music server running fidelizer windows 8 audio optimizer
    PS Audio Quintet power center

  5. #5
    Suspended Smokey's Avatar
    Join Date
    Jan 2004

    Thanks everybody

    I also been following CR since 80's, mainly their TV ratings. They have been pretty consistent in accuracy of their ratings, but ratings is not always 100% correct all the time. And most of it can be attributed to TV's own variations and performance differences within a model number.

    Quote Originally Posted by TheHills44060 View Post
    AVSForum is awful.
    Members can be get little bit cocky, but there also alot of good information and knowledgable poeple on that site...such as editor of CR Jim Wilcox

  6. #6
    Forum Regular
    Join Date
    Nov 2013
    It was a nice visit to TV Testing Lab.

  7. #7
    AR Newbie Registered Member
    Join Date
    Dec 2013
    Also have a B&O System from the 1980s, a Carver System from the 1980s, and an 80s Revox system.
    Go through the brain dumps design tutorial and PMI guide to learnuseful web design skills. Become expert in web design with Cornell University online course. for more details click here wikipedia you can find us on Facebook

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts