Consumer Reports and Speaker Ratings

[thekid]

Recently I found several old copies of Consumer Reports (2001-2003) that contained the ratings of several types of speakers. They state their ratins/rankings are based on how "accurately" the speaker reproduces sound. After reading this forum several of the speakers/manufacturers that held in high or low regard here are almost polar opposite of what Consumer Reports rating/ranking. I know CR is not a sterophile magazine but they appear to taking a scientific approach to reporting a speakers ability to reproduce the original source. Can anyone explain/comment on the value of Consumer Reports as a useful source for determining future purchases. Sorry if this has been threaded before but I thought I'd ask...

[RGA]

They don't listen to the speakers -- I find nothing overly scientific about their evaluations do you. A speaker's job is to reproduce music in a satisfying manner over a long period of time...A test that does not incorporate this into it's parameters is not useful.

On the other Hand -- Stereophile and big Hi end magazines reviewers are not necessarily trustworthy -- your ears are more trutworthy when shopping -- why not trust them at least a little bit.

Consumer Reports does not provide any real useful information -- though I sometimes like their repair history notes etc on cars and electronics. I think Audio is tougher to reduce to a spec sheet than say a printer is. But many will no doubt disagree.

[thekid]

I agree with the trust the ears advice. I guess I hear a lot of technical comments made in this forum and was wondering how their could be such a divergence of opinions if a speakers accuracy is something that can be measured precisely. I can't comment on the method CR uses. They are as you say a good source for auto ratings and other products so tend to think their methodology as a solid basis behind it. But when I Bose for example, ripped as poor speakers in this forum but near the top in some of the CR tests I just wonder where the disconnect occurs. Based on threads I have read and if I understand you correctly that a speaker can fade/lose accuracy over time and the CR testing method is just a one-time snap shot and does not reflect the qualities of a speaker over time?

[theaudiohobby]

Recently I found several old copies of Consumer Reports (2001-2003) that contained the ratings of several types of speakers. They state their ratins/rankings are based on how "accurately" the speaker reproduces sound. After reading this forum several of the speakers/manufacturers that held in high or low regard here are almost polar opposite of what Consumer Reports rating/ranking. I know CR is not a sterophile magazine but they appear to taking a scientific approach to reporting a speakers ability to reproduce the original source. Can anyone explain/comment on the value of Consumer Reports as a useful source for determining future purchases. Sorry if this has been threaded before but I thought I'd ask...

If you are interested in the definitive commentary on their correlation or lack of, I suggest you read some of Dr. Floyd's O'Toole comments and their associated AES papers, one of their studies says

Among the results are conclusions that measurements with1/3-octave resolution are not adequate, that sound power or in-room measurements alone are not sufficient to predict listener preferences, and that the flatness and smoothness of high-resolution on-axis curves need to be given substantial weighting.

On a side note, a lot of folk's impressions about scientific based testing is based on what they read in consumer magazines and is really the wrong place to look, they are consumer orientated and rigorous scientific testing is simply beyond their scope, certain parameters are simply not measured due to cost or brevity. However a knowledgeable eye can still learn a great deal about a speaker behaviour from some of the detailed treatments.

[Pat D]

If you are interested in the definitive commentary on their correlation or lack of, I suggest you read some of Dr. Floyd's O'Toole comments and their associated AES papers, one of their studies says



On a side note, a lot of folk's impressions about scientific based testing is based on what they read in consumer magazines and is really the wrong place to look, they are consumer orientated and rigorous scientific testing is simply beyond their scope, certain parameters are simply not measured due to cost or brevity. However a knowledgeable eye can still learn a great deal about a speaker behaviour from some of the detailed treatments.

Consumers Reports had the advantage of providing objective data but not enough to be very useful. One criticism has been that they did not take directivity into account and this was made long ago by none other than Julian Hirsch. As well, as you point out, the research has shown that 1/3 octave resolution is insufficient and that 1/10 octave or better is needed. Floyd Toole's comments on CR reports can be found on page 20 of the White Paper you linked:

http://www.harman.com/wp/pdf/AudioScience.pdf

[RGA]

The attack on Bose does continue -- but for interest sake Hi-fi Choice did blind level matched panel listening sessions and awarded a recommendation for a Bose speaker back in the mid 90's the 305 I believe was the number. Julian Hirsch said the 901 was the best speaker ever built -- and a lot of folks put a lot of stock into what he said.

Hi exact words were "Nevertheless, at this moment, I must say that I have never heard a speaker system in my own home which could surpass or even equal the Bose 901 for overall "realism" of sound."

And they have technical white papers out of MIT telling you why nothing is as good and so does Harman -- get the speakers in the same room and listen to them. This is not hard. They have to first work in real world listneing rooms not ones made out of foam walls. Then they have to work in your room. I have heard the 901s in about 5 rooms and every time I've felt they were very poor as I have felt with every Bose I've heard. But it's usually the price that kills them more than the sound. If they were about 1/5 the price they'd be competitive. But as Bose has proven -- they don't need to do anything to change the formula - put the technobabble get a few famous reviewers on board -- and you will meet your sales quotas. The Bose model is one most have tried to copy -- plenty of marketshare to aqcuire.

[thekid]

I think I may have started another Bose thread here....Re Woochifer's comments.. If I understand you correctly you are saying is that CR does not weight their test it basically is pass/fail. So if I speaker for example can not hit highs but does everything else right it might be deemed accurate by the CR methodology. So it really becomes a question of hearing what is missing or finding what you like correct? But would'nt the CR tests have some value in the sense they would tell you that a speaker that is rated accurate in their tests might have deficiences but they would have less deficiences than a speaker rated lower. That would help narrow your search somewhat but you would still need to find the speaker that best suits your personal preference.

At the risk of fueling the lurking Bose thread here....I am not exactly buying the lawsuit=good ratings statement stated/hinted in a couple of threads. Automobiles are more my field of knowledge and if several auto manufacturers who have not been rated highly by CR over the years thought that lawsuits were a way to affect CR's opinions they have teams of lawyers who could bury CR with lawsuits. Certainly GM has deeper pockets than Bose and if the tactic described were effective that would have used it years ago. If CR could so easily be intimidated they would have been out of business years ago.

[Haru]

Julian Hirsch said the 901 was the best speaker ever built -- and a lot of folks put a lot of stock into what he said.

I have a pair of "The best speaker ever built" The bose 901 series II's

I would be glad to part with the "The best speaker ever built" for a nominal price.

Seriously, The 901's need proper room placement and amplification in order to sound the way it was designed to sound. If set up correctly, they are great for background music or rear surrounds if you have enough horsepower to get them up and running.

Please no offense intended to those of you who are running 901's for your daily drivers. Its just one man's opinion

haru

[Pat D]

The attack on Bose does continue -- but for interest sake Hi-fi Choice did blind level matched panel listening sessions and awarded a recommendation for a Bose speaker back in the mid 90's the 305 I believe was the number. Julian Hirsch said the 901 was the best speaker ever built -- and a lot of folks put a lot of stock into what he said.

Hi exact words were "Nevertheless, at this moment, I must say that I have never heard a speaker system in my own home which could surpass or even equal the Bose 901 for overall "realism" of sound."

And they have technical white papers out of MIT telling you why nothing is as good and so does Harman -- get the speakers in the same room and listen to them. This is not hard. They have to first work in real world listneing rooms not ones made out of foam walls. Then they have to work in your room. I have heard the 901s in about 5 rooms and every time I've felt they were very poor as I have felt with every Bose I've heard. But it's usually the price that kills them more than the sound. If they were about 1/5 the price they'd be competitive. But as Bose has proven -- they don't need to do anything to change the formula - put the technobabble get a few famous reviewers on board -- and you will meet your sales quotas. The Bose model is one most have tried to copy -- plenty of marketshare to aqcuire.

Julian Hirsch's review of the original Bose 901 was published way back in September 1968, a fact you somehow neglected to mention. You can follow the link here and click on his review and no, he did not say it was the best speaker ever made, though he was enthusiastic about it.

http://www.soundandvisionmag.com/art...article_id=536

Very few speakers available in 1968 would compare very well with good speakers nowadays (which is not to say the would sound bad or unpleasant). Some would, especially since the advent of computer aided designs in the early 1970's. But then, have you ever actually heard the early series 901's? I have and they weren't bad sounding at all as I remember.

Oh, I understand! This is another roundabout way to get a dig at Harman International and its highly respected Vice President of Acoustical Engineering, Dr. Floyd Toole, who had nothing to do with the development of the Bose 901. You're really stretching with this one.

[RGA]

Patd

He said nothing surpassed it or was the equal to it ---- what does that say to you? It says to me nothing is is better and nothing is as good -- which means it is the best.

1968 -- please there were good speakers abound.

Yes I heard the original ones and two newer sets...and all were no good.

In fact I don't know much about him just that his name gets mentioned -- Until you said it I didn;t know he worked for Harman...but since all the Harman speakers and copy cats I've heard over the last 15 years -- well it's not surprising to me that he likes Bose!

[shokhead]

Julian Hirsch's review of the original Bose 901 was published way back in September 1968, a fact you somehow neglected to mention. You can follow the link here and click on his review and no, he did not say it was the best speaker ever made, though he was enthusiastic about it.

http://www.soundandvisionmag.com/art...article_id=536

Very few speakers available in 1968 would compare very well with good speakers nowadays (which is not to say the would sound bad or unpleasant). Some would, especially since the advent of computer aided designs in the early 1970's. But then, have you ever actually heard the early series 901's? I have and they weren't bad sounding at all as I remember.

Oh, I understand! This is another roundabout way to get a dig at Harman International and its highly respected Vice President of Acoustical Engineering, Dr. Floyd Toole, who had nothing to do with the development of the Bose 901. You're really stretching with this one.

Good speakers in 68 can be as good as equal good speakers today. IMO,speakers havent changed much in the line of A/V.

[Woochifer]

Problem with CU's testing methodology is that their accuracy ratings are based on statistical deviations from a flat frequency response. Inaccuracies in the midrange are treated identically to inaccuracies in the highs or lows, where considerably less of the sound information from movie or music soundtracks goes. Plus, they make a special accommodation in their testing for Bose speakers because of their direct/reflecting design. This (and the numerous lawsuits that Bose has filed against Consumer Reports) might explain how Bose has lately been rating so high on the CU ratings.

[bjornb17]

CR pretty much gives every Bose product an outstanding rating, while other speakers that are actually good somehow get lesser ratings.

[kexodusc]

1/3 octave resolution? That freakin' curve will look nothing like it's true self. These guys don't even listen to the speakers? Well, that might not be such a bad thing, since 10 evaluators still don't have my ears, but c'mon...
I had no idea these guys were that bad...why bother?

[nightflier]

...the numerous lawsuits that Bose has filed against Consumer Reports) might explain how Bose has lately been rating so high on the CU ratings.

I've sent a quite a few letters to CR regarding their A/V reviews. Generally speaking, they have a limitted budget and they may not consider speakers to be all that high on the priority list because they tend to emphasize features over performance. I have a friend who repairs refrigerators and he pretty much has the same complaints about what he know best.

I do have a wish list if anyone is going to mail them about speakers (I think they are tired of hearing from me):

- Include construction, bracing, materials, and durability of grills and finish as critera.
- Include tests for extreme conditions like how well does the speaker stand up to very loud levels and what happens when it bottoms out. Also: is there a hum at very low levels?
- List what equipment they are using for testing (what they list is too vague most of the time)
- Do customer surveys over time for durability and quality.

Regarding the selection of speakers, they will only test speakers that are easily bought at several locations, so most internet-only companies are left out. They also test lower priced speakers so that means the low end of the better companies and the rest of the consumer-oriented stuff at CC, BB, and GG, and even the dreaded "-marts." and Costco. After all, they are called "Consumer" Reports, right?

Regarding the Bose lawsuits influencing the results, I doubt that. They have stood up against the dairy industry, automakers and software companies, and so I doubt that Bose is a big threat. Bose does sue them quite often (and everyone else, for that matter), but seldom succeeds. I would rather think that the high ratings for Bose is the result of meager testing criteria, budgets, and manpower.

They do respond to your letters, so write them!

(And, no I don't work for CR)

[thekid]

Thanks for coming back on topic....
I agree with your list and will see if I can get them to address them. As I have re-read their tests they really do stay to the basics and do not seem to elaborate. A friend me showed some issues back in the 90's and they actually had some optimum settings and other details that are now absent.
Hopefully as HT becomes more common place they start testing some higher end speakers and upgrade their testing/reporting methods.

As I mentioned in an earlier post I don't buy the oft repeated lawsuit Bose bash either. There a lot of other companies with deeper pockets than Bose who have suffered at the hands of CR over the years and would love to be able to use a lawsuit or threat of a lawsuit to get a positive review. Maybe "Myth-Busters" can dedicate a show to it....

[John Ashman]

Kid,
Here's how CR measures speakers. This is from memory of their own description a few years back, but I believe it's correct. They take ~16 measurements from in front, to the side, above, behind, etc. Picture a sphere with multiple points. Those points are averaged. Then, in the most inexplicable part, they take the graph and spin it clockwise or counter clockwise, which ever makes the line more horizontal "in order to mimic the effect of a theoretical tone control". When Paul Barton, who worked at the CRC in Canada was visiting my store, I asked him about that, and he said that CR folks visited the CRC and he took them on a tour and he finally asked him how THEY measured speakers and he told them, as I recall "Uhhhh, you know that's wrong, don't you?" and he said they told him "Well, yes, but we have to have something that's objective". And that's how CR comes to it's entirely screwed up, meaningless conclusions. Moreover, unless you measure distortion, spectral decay, impulse response, etc, you're only getting a really BAD picture of ONE aspect of performance. Completely useless. Which is why CR is one of the most DANGEROUS magazines on the planet.

[thekid]

Thanks for the detailed reply. I'm learning.......slowly but I'm learning.