What would make HiFi Reviews (more) useful?

[Ajani]

A recent discussion with RGA in a Speaker thread reminded me, that I had planned to start this thread almost 2 weeks ago... However, a case of severe laziness has prevented me from doing so until now... So here goes:

What would make HiFi reviews (more) useful to you?


Objective versus Subjective?

Some of us younger audiophiles/enthusiasts weren't aware that before J. Gordon Holt launched Stereophile all HiFi was purely about objective measurements... No one felt the need to actually listen to equipment... You just ran test measurements and recommended whatever measured best...

Reviews based solely on Listening to the equipment are open to sighted bias. But reviews based solely on measurements assume that what we hear is solely determined by the items we know how to measure. (Scientists were convinced the world was flat until someone proved it was round - so just because we don't know how (or what) to measure yet, doesn't mean we won't find it out one day...

Advertising or Subscription only?

For major HiFi mags like Stereophile (US) and What HiFi? (UK), the editorial staff is seperate from the advertising team... So John Atkinson of Stereophile doesn't get to see ads that appear in the mag before you and I do, and he has no input in the advertising process...

However, the prevailing school of thought seems to be that HiFi mags are slaves to their advertisers and that negative reviews are never published (which is patently false and easily disproved by anyone with a desire to do so)... So would having all mags turn to subscription only and drop all advertising be the answer?

Also, would you be willing to pay 5-10x the current price of the HiFi mags for an advertising free version?

Lots of Negative Reviews or Mostly Positive Ones?

Here in the western world, we see far more positive reviews of products, than in the UK (where nasty reviews occur frequently)... The reason for so many raves here is because reviews mostly select products they are interested in to review, so odds are that they will like most of what they review. In the UK, reviewers/review teams pretty much review anything they can get their hands on. Also the UK tends to give ratings 1 - 5 stars or globes, while the US usually just writes a long article. So which one is better?

In the US, you see far less really nasty reviews, but in the UK, those nasty reviews are very often contradicted by other mags... So one mag well rate a product with 5 out of 5 stars and name it product of the year, while another will give it 3 stars and say they can't even recommend it...

To Shootout or not to Shootout?

UK Mags tend to do regular shootouts of products in the same price range, while US Mags generally don't. Is a shootout more or less useful than a solitary review?

Much like with negative reviews, shootouts in the UK usually contradict each other...

I'll add other areas for discussion later (assuming anyone is interested in this topic)... But feel free to share your own views...

[Ajani]

the review industry is highly problematic due in part to the structure of the magazines and the honesty of the individual reviewers.

While I don't agree with Arthur Salvatore on all points he does raise some issue that should be considered when buying into review magazines.

Take the recommended component listing of Stereophile

Under Holt in 1985 there were 9 class A products (combined) in 2000 under Atkinson there were 104 class A recommended components.

"Please consider these statistics carefully. As for myself, I focused on just two obvious and highly relevant details:

In the Summer 1971 issue, there were NO advertisements and 7 components were in "Class A". In the Fall 1985 issue, more than 14 years later, there were still only 9 components in "Class A", despite going from 0 to 47 pages of advertising. However...
J. Gordon Holt was still the Editor during all that time.

Then John Atkinson arrived on the scene.

In short order, there were profound changes, starting from the late 1980's and continuing through the entire 1990's. By April, 1992, there were already 30 components in "Class A". This was just a "warm-up"...

By October 2000, 14 years after Atkinson's arrival, there were 104 components in "Class A". Could there be any "innocent explanations" for this obvious trend? Well, three "excuses" have been used.

Excuse No. 1

The performance of today's components has improved (or "advanced") on those of the past. Assuming that this is a fact, doesn't that mean more components should in "Class A"?

Answer: NO!

The fact that there were numerous "advancements" is totally irrelevant. This is because each and every new advancement must automatically supercede the previous advancement, or else it wasn't an "advancement" in the first place.

As each new improvement "raises the bar" to get into "Class A", any older model, which can not reach that new "bar", is relegated to "second best", which means they can no longer honestly remain in "Class A", which is supposed to be "the best attainable sound" at that time. Just as the newest, fastest computer chips relegate the older chips to "second fastest or best". Ruthless logic yes, but true when you are talking about "the best".

This principle is the primary reason why all of the numerous, earlier advancements during "The Holt Era", from 1971 to 1985, did NOT result in an increase in the "Class A" recommendations.

Excuse No. 2

There are more components available now than ever before. Doesn't that mean that more components should be in "Class A"?

Answer: NO!

The best is the best, no matter how many "participants" are competing for that "title".

An Example: There were far more competitors at the 2008 Summer Olympics "than ever before", but there were still only 3 medals given out for each event. In pro sports, there is just one "all-star team", no matter how many expansion teams and new players are added.

Stereophile, between 1971 and 1985, faced a huge, relative increase in the available number of components (plus the advent of accepting advertising). Even so, during this entire 14 year period, under J. Gordon Holt's direction, Stereophile went from 7 to only 9 "Class A" components.

This historical fact is the final proof that there is not any "law" or "rule" that the Editor must increase his "Class A" recommendations just because there are a greater variety of components.../...

"Class A"?

In October 2000, there were 46* amplifiers alone in Class A, the so-called "best attainable". There are still dozens as this is written. Only someone who is intellectually dishonest, in every sense of that expression, can claim there are 46 "best" anything's. (Do you know anyone with 46 "best friends"?) All the other Class A component categories have had similar, totally implausible expansions.

Atkinson even created a new Class, "A+", which is even better than "the best"! In all human history, and in all human cultures, it has been philosophically impossible to be better than the best, except in Stereophile. It's not even a rare occurance. In fact, in their April 2003 RCL, there were more Digital Processors in Class A+ (7), than in Class C (2)!

*During the publishing control of J. Gordon Holt, from 1962 all the way to the middle 1970's, the highest number of amplifiers in Class A was 4. The lowest number of amplifiers in Class A was 1. Holt kept only that one single amplifier in Class A even after it was discontinued. This means Holt refused to place even one unworthy component into Class A, because he understood and respected the true meaning of the word "best". Now compare Holt's intellectual integrity to that displayed by John Atkinson." http://www.high-endaudio.com/RR-STEREOPHILE.html

Now I want to say that I don't agree with Arthur on his attacks of Atkinson (and there is a heavy dose of unsubstantiated and unfair attacks) who very may well feel that such league tables are valuable - it is not dishonesty to have a different view than the guy you replace. Nevertheless, I do agree that having so many "class A" products and rave reviews dilutes what truly is the best of the best.

I think that such a league table would or should operate as a bell curve such that the best of the best would be in the smallest percentile - that's what best is supposed to mean.

Even then it is still just opinion - the best thing to do is find a reviewer with a similar ear to you. I like HE and SET based systems more than big power low efficiency systems and I believe a system should be able to play all music because the stereo should not care what is being played - it's job is to reproduce it. Others do not agree with that assessment and so they should find other reviewers who may offer them better help.

Audio reviewing is not all that different than movie reviewing. No matter what your favorite critic says - you're not going to agree 100% of the time.

Mike at AudioFederation and I agree 100% on Audio Note speakers and systems and many other rooms at CES but he HATED the Sony speakers and didn't like Magico speakers - I felt they were some of the better rooms at CES. The Sony just about made my top 5. So even though I trust Mike's advice - there are some polar opposite views.

I like Gordon's elite listing and I think that over time I may develop such a scheme of ranking. I went to over 70 rooms at CES and I could point to maybe 3 loudspeakers that I felt were true standouts. Then maybe another 10 that were excellent but a step down and then another group of good quality sound but didn't do it for me perhaps at the prices they were charging.

I wanted to address this point from RGA in another thread:

While I don't agree with Arthur Salvatore's attacks on John Atkinson (I think they are unsubstantiated) I agree that it is a good question of whether having so many 'Class A' components is actually useful... According to Stereophile, My Benchmark DAC1 is Class A, but the Bel Canto DAC3 and the Logitech Transporter are both supposed to be slightly better than it... What??? So the real question becomes what do they mean by Class A? Is Class A the best of the best (as Holt clearly had it) or just top quality HiFi? So maybe Holt regarded Class A as getting a 99% in a test, while under the Atkinson Admin, Class A means getting an A (90% to 100%)...

Another factor to consider it that when Holt was the only reviewer, he alone selected Class A, now John Atkinson does not select all Class A components. A product is rated Class A because at least one Stereophile reviewer regards it as being Class A (even if others disagree)...

[JoeE SP9]

Maybe Class A+ is when two or more reviewers agree on a Class A rating for a given product.
I started reading Stereophile in the early 80's when JGH was doing the rating. I too think that there is something terribly wrong with all the class A products. Everything and a bag of chips can't be the best! The Class A+ rating is just plain silly.
Stereophile should cut the list of recomended components by at least half, maybe even 75%.

[RGA]

I have started a discussion with my fellow reviewers at dagogo about a similar problem that pervades the review industry.

To me the stereophile class rating system is highly problematic but most such systems are compromised - probably why Constantine has so far avoided them.

No one single person can review every product so how do we come up with a best of listing. A reviewer can vote on it and submit it to the editor but in the end it is the editor's choice.

Take Art Dudley of Stereophile - he reviewed the Audio Note AN E/SPe HE - and he bought them as his reference speakers. Stereophile rates it as a class B loudspeaker even though Dudley says they should be an A. (I believe his words were should be an A+++)

So who made the decision? Did other reviewers agree? Well Wes Philips recently said (Jan 2009) that a room with an AN E loudspeaker was not only the best sound at CES but the best sound he has EVER auditioned. That's a second Stereophile writer - and these two guys are some of the top reviewers at the mag. A third writer there owns an AN E (Peter Van Wellinswaard who is a technical adviser) and John Marks called the AN E one of the most memorable experiences of his life.

What's more important - a reviewer who puts his own money out and buys them to use in his own house or something that is given a class A rating that not a single reviewer would touch with a ten foot pole? Audio Note does not advertise in Stereophile and John Atkinson and Peter Qvortrup don't see eye to eye on much. So I suppose it is amazing it got a class B - though PQ showed up at CES this year and you will notice they didn't cover the room - coincidence?

I think Arthur has a point about the rating systems with his analogy to the olympics. Even if stuff has improved there should be a league table where there is not more class A gear than class C - it makes no sense. Unless of course they are saying that it all sounds the same - in which case there are serious problems.

Notice that recently Stereophile gave very negative reviews to a Bryston and a Totem - the next issue did not have the usual advertising from either company.

To me there are so many problems with recommended component listings because they're highly arbitrary. Not everyone is on the same page at the same magazines so how on earth do they come up with such lists?

My list would be very different from Doug Schroeder's or Jack Robert's, or Fred Crowder's for example. I suppose the ones where we all agree that ABC speaker is very good but even then there are degrees of preferences.

The same problem pervades magazines that give out yearly awards - so if XYZ speaker wins best loudspeaker in 2010 how often do you see it win again - or is the award given every year just to have something new in the limelight. If I were giving out reviews I may have the same product win year in and year out until such time as something proves to me that it is better. That may happen but I never see a product win over and over and over. Stereophile though dumps them from the list after a few years or if they're no longer sold - that is a problem too. The assumption is if it is new it is better - it ain't so.

And of course as massive a problem as all that - what about the stuff that Stereophile misses? For yeard they were recommending amps in the Class B such as my old Arcam Delta 290p but no one there reviewed the classic Sugden A21a which is a much better amplifier. Finally, after it's 30th+ year or so selling them they bother to get around to it and Marks felt it was the best sounding integrated for the money. So while they're out recommending good but not star budget integrateds to the thousands of people buying the magazine - the entire time they WOULD HAVE recommended the Sugden instead.

A smaller magazine has an excuse to miss a staple product - but the longest selling integrated amplifier on the planet is missed by Stereophile?

None of this would be problem if most of the readers understood these things and didn't put much stock in such ratings but I get the feeling that a great many people do put a LOT of stock in them. They believe if they shop off the class A list they will have a great sounding stereo. Some of the stuff that I have heard on the class A list is truly shockingly bad IMO. I think to myself - something is really odd - nobody working there bought them.

In the end it comes down to either entertainment or finding a reviewer you agree with - has a similar ear and values certain things you value. that is why so much stuff gets raves - they will simply give the panel to the panel guy, the SS to the SS guy the tube to the tube guy. Everything gets a rave. What you want is if you are the tube guy is for the reviewer to review two tube amps at around the same price and tell you the pro's and cons of each. But manufacturers often don't like that.

[Ajani]

I have started a discussion with my fellow reviewers at dagogo about a similar problem that pervades the review industry.

To me the stereophile class rating system is highly problematic but most such systems are compromised - probably why Constantine has so far avoided them.

No one single person can review every product so how do we come up with a best of listing. A reviewer can vote on it and submit it to the editor but in the end it is the editor's choice.

Take Art Dudley of Stereophile - he reviewed the Audio Note AN E/SPe HE - and he bought them as his reference speakers. Stereophile rates it as a class B loudspeaker even though Dudley says they should be an A. (I believe his words were should be an A+++)

So who made the decision? Did other reviewers agree? Well Wes Philips recently said (Jan 2009) that a room with an AN E loudspeaker was not only the best sound at CES but the best sound he has EVER auditioned. That's a second Stereophile writer - and these two guys are some of the top reviewers at the mag. A third writer there owns an AN E (Peter Van Wellinswaard who is a technical adviser) and John Marks called the AN E one of the most memorable experiences of his life.

I guess the AN E must have been reviewed before Stephen Mejias started compiling the recommended components list... Since his policy that is stated in all the current rec listings is that Class A is based on one reviewer regarding the product as Class A... So the AN E should be Class A if Art Dudley thinks so...

What's more important - a reviewer who puts his own money out and buys them to use in his own house or something that is given a class A rating that not a single reviewer would touch with a ten foot pole? Audio Note does not advertise in Stereophile and John Atkinson and Peter Qvortrup don't see eye to eye on much. So I suppose it is amazing it got a class B - though PQ showed up at CES this year and you will notice they didn't cover the room - coincidence?

I think Arthur has a point about the rating systems with his analogy to the olympics. Even if stuff has improved there should be a league table where there is not more class A gear than class C - it makes no sense. Unless of course they are saying that it all sounds the same - in which case there are serious problems.

Notice that recently Stereophile gave very negative reviews to a Bryston and a Totem - the next issue did not have the usual advertising from either company.

To me there are so many problems with recommended component listings because they're highly arbitrary. Not everyone is on the same page at the same magazines so how on earth do they come up with such lists?

My list would be very different from Doug Schroeder's or Jack Robert's, or Fred Crowder's for example. I suppose the ones where we all agree that ABC speaker is very good but even then there are degrees of preferences.

The same problem pervades magazines that give out yearly awards - so if XYZ speaker wins best loudspeaker in 2010 how often do you see it win again - or is the award given every year just to have something new in the limelight. If I were giving out reviews I may have the same product win year in and year out until such time as something proves to me that it is better. That may happen but I never see a product win over and over and over. Stereophile though dumps them from the list after a few years or if they're no longer sold - that is a problem too. The assumption is if it is new it is better - it ain't so.

And of course as massive a problem as all that - what about the stuff that Stereophile misses? For yeard they were recommending amps in the Class B such as my old Arcam Delta 290p but no one there reviewed the classic Sugden A21a which is a much better amplifier. Finally, after it's 30th+ year or so selling them they bother to get around to it and Marks felt it was the best sounding integrated for the money. So while they're out recommending good but not star budget integrateds to the thousands of people buying the magazine - the entire time they WOULD HAVE recommended the Sugden instead.

A smaller magazine has an excuse to miss a staple product - but the longest selling integrated amplifier on the planet is missed by Stereophile?

None of this would be problem if most of the readers understood these things and didn't put much stock in such ratings but I get the feeling that a great many people do put a LOT of stock in them. They believe if they shop off the class A list they will have a great sounding stereo. Some of the stuff that I have heard on the class A list is truly shockingly bad IMO. I think to myself - something is really odd - nobody working there bought them.

In the end it comes down to either entertainment or finding a reviewer you agree with - has a similar ear and values certain things you value. that is why so much stuff gets raves - they will simply give the panel to the panel guy, the SS to the SS guy the tube to the tube guy. Everything gets a rave. What you want is if you are the tube guy is for the reviewer to review two tube amps at around the same price and tell you the pro's and cons of each. But manufacturers often don't like that.

A major part of the problem with recommended components lists (or any kind of product of the year awards or even reviews in general) is that consumers don't understand how to use them...

People just assume that if you combine 3 products of the year or 3 Class A components, you will get a great sounding stereo... Which can be a very costly mistake...

Despite the fact that many mags openly state how to use (and not use) their recommended products listings, so many persons either don't read or just ignore those explanations... It's a tough call for the mags....

[Ajani]

If a reviewer buys a product below retail, can he be a judge of its value for money?

I know that (as RGA mentioned) the best recommendation a reviewer can make is to buy the product he reviewed... But can someone who buys a product for 40% or 50% of MSRP, give a credible opinion on its value for money? There are many products that persons are willing to buy used on Audiogon, but that they don't think are worth retail price...

So if a reviewer bought a $10K Speaker for $4K is that really a recommendation that we should buy the speaker at $10K? I've heard the argument that the reviewer could purchase any $10K Speaker he reviews for that price, so the fact that he bought the particular model means it is his favorite $10K speaker... That I have no issue with... But the question I pose, is whether the reviewer would still buy that Speaker if he had to pay retail ($10K) or would he think that the difference between it and a Speaker retailing at $4K is not that great?

Assuming he can buy any product at 40% of list, then a $4K Speaker would cost him $1.6K and a $10K speaker would cost him $4K... The difference between them being $2.4K... If he had to pay list, then the difference between them would be a far more substantial $6K, so would he still think that $10K speaker was really worth the extra outlay over the $4K one? Would he still buy it?

[RGA]

I happened to read the Stereophile recommended list and it stated that Art felt it deserved class A++ or something and it went to class B. That was in 2009 so I am not sure what is going on there.

As for reviewer's discount - I can't speak for them all - I have not asked about any discounts. But in theory if we can get say 50% off then that would be typical across the board so all manufacturers are equal - so we would still choose our favorite product within our budget just like anyone else. In most cases I believe you could do better than a reviewer's price simply by buying it used. The McIntosh MA 7000 is $8800 new - I might be able to buy it for $4,000 maybe even $3000 with reviewer discount - my dealer sold a mint shape one used for $2000. This is a current model. So I don't consider discounts to be much of a plus - I have paid far more out of my own pocket being a reviewer.

As for value for money - this is always so difficult to answer. In my show report I try to do this by selecting my five favorite rooms with speakers under $10k (based on the speakers because generally less expensive speakers will have less expensive front ends) and then rooms with speakers over $10k.

I feel that most audiophiles who take this seriously will get to the point where they are spending in the $6-$10k price range for a set of loudspeakers. I managed the AN J going to university with no job and living on student loans - I chose not to own a car because audio was more important to me. So it can be done. I also bought the OTO during that time.

One thing that you raise is the reviewer's system has a great deal to do with "perceived" credibility. If you are a manufacturer selling $80,000 amplifiers who would you rather send your amplifier to. Richard Austen who owns a $4k amp and $5k speakers or Fred Crowder who owns $192,000 speakers and $95,000 amplifiers? Both of us may be just as good at auditioning gear and evaluating strength and weakness etc. And I may have auditioned the same ultra expensive amps - but if you are selling $80k amps you will send it to Fred.

I am on the one hand jealous that he gets to play with the expensive stuff - but if I did I might be heartbroken to have to say goodbye to it. Besides - I figure 80% of the readers are in my general budget zone so by reviewing the affordable stuff I might get read more. So there are always advantages - besides $80k amps ought to be darn good - It's more fun to find those $5k speakers that are truly wonderful.

I can't speak for others but I try to look at the price performance ratio of everything I review. The show report is a little different - I comment on the sound regardless of price. The Perfect8 Technologies speakers are $375,000 and they sounded very very good. There is no way on earth that what I personally heard justifies the $375,000 price tag. But this is a sculpted piece of art that are catering to a certain clientele and the technology is there the look is there and the sound is very good. If one has the pockets to pay that price they don't really care what I have to say about them in terms of price performance.

Vandersteen has a house sound you either like or you don't. If you like them and you want to keep going up the Vandersteen line then perhaps you would find their top $45k model worth it to you. I liked the sound of them I must say but I would sooner buy a Teresonic or Audio Note for under $10k because to me they simply sound more natural and right. In fact it's not much different than spending $51k on the AN E Sec Sig - if you get the sound of the AN E then it is possibly the only speaker for you and you may decide to continue up the line to $50k versions.

The other issue to go along with what I said about Fred is that I can't really look at a $50k speaker and say that it's good or bad value in absolute terms. The Vandy at $45k is their best speaker - it sounds better than all the others. Is it 10 times better than the $4k ones - no but then times better isn't the point some people are willing to pay ten times the price for a 5% improvement.

Still I think I can say that when I hear certain very expensive products that I can say "wow that truly is the best I have heard" or "it really does scale like little else". The Acapella High Violencello II with Plasma Tweeters at $80,500 is a speaker I will never own unless I win a lotto.

Of course the price seems absurd - this is not even on of their 5 most expensive - they go up over $800k. But I can safely say that it was truly stunning - It's a statement loudspeaker in a sense of transients on drum sticks that is exceptional. If you have the money they're worth it - and if you don't then who cares if it is worth it or not - it's moot based on your income.

I think in the end too much is made of money spent. IMO the AN E/Spe HE is a speaker that can be the end of the road kind of speaker - it's $7,600 and it stands with anything I heard at CES regardless of price. Yes there is better - certainly better in isolated traits - but at silly prices based on my savings account. I liked them a LOT more than most of the $15,000 - $70,000 loudspeakers I was hearing. I could also say that about Gallo's new 3.5 at a modest $5,900 or KingSound's Prince II at $6,000 - or splurge and get the $8k Kings.

You can spend stupid prices and occasionally get the results but most of us are in the sub $15k camp for speakers. If I had the money I would buy the AN E/Spe HE. But I would also buy any of the speakers in my final 5 at the end of my show report as alternatives based on those auditions and I think they are worth the retail price - in some ways I find them all a little under priced based on the sound of the market at CES.

[E-Stat]

Objective versus Subjective?

Wrong comparison. It is "objective" vs. "observational". To observe that a condition exists exists involves no subjectivity. Subjectivity occurs afterwards in either case with a judgment regarding that change. Two people can agree completely on a difference and yet arrive at different conclusions. I've known two prominent reviewers for decades and sometimes disagree with either as to the subjective side of what I find best - having heard exactly the same things.

You just ran test measurements and recommended whatever measured best...

Those of us who do remember that approach by Len Feldman and Julian Hirsch recall how utterly useless they were to convey any useful knowledge as opposed to information.

Reviews based solely on Listening to the equipment are open to sighted bias.

Unfortunately, the alternative is to employ switch boxes which using their common grounds (or cause horrible switching thumps) end up comparing both components to both. Theory and reality do not converge. As with cables, many attempt to look at the audible effect of the box in a vacuum and assume their impact while in circuit is identical - when it is not. When you examine their electrical effect in the system, however, you find a different situation. Frank Van Alstine pointed this out long ago.

But can someone who buys a product for 40% or 50% of MSRP, give a credible opinion on its value for money?

In the comparative sense, sure. Remember too that reviewers (and store employees) can get any component at accommodation (I've purchased a couple myself that way), so it is a level field when comparing one to another.

I think TAS should return to its roots with the approach HP took long ago before he had to turn the reins over to new management.

rw

[Feanor]

Arrrgh ... just ... can't ... resist ... plug for AN ...

....

Take Art Dudley of Stereophile - he reviewed the Audio Note AN E/SPe HE - and he bought them as his reference speakers. Stereophile rates it as a class B loudspeaker even though Dudley says they should be an A. (I believe his words were should be an A+++)

So who made the decision? Did other reviewers agree? Well Wes Philips recently said (Jan 2009) that a room with an AN E loudspeaker was not only the best sound at CES but the best sound he has EVER auditioned. That's a second Stereophile writer - and these two guys are some of the top reviewers at the mag. A third writer there owns an AN E (Peter Van Wellinswaard who is a technical adviser) and John Marks called the AN E one of the most memorable experiences of his life.

....

RGA's AN E/SPe HE wet dream.

[Feanor]

I never trust a single review, not even from a reviewer with a good record in my own books. I look for corroboration amongst reviews, and not just all "thumbs up" but expressing a similar impression, explicitly or between the lines.

I do always enjoy a comprehensive technical description of the component, even if I don't always understand all the details. (In this regard I enjoy 6 Moons reviews for example.)

I do like objective information though I don't rely on it. For one thing, objective reviews are constrained to a certain uniformity of testing that doesn't necessarily reflect set up requirements the equipment, especially speakers.

So for example when testing some AN speakers John Atkinson didn't set them up in room corners as AN prescribes for best results. So what would the measurements have been if they were set up as specified? We don't know. Atkinson justified his approach on the basis that he test are speakers in a certain way to ensure fair and consistent testing -- that's objectivity, right??

[mlsstl]

I believe many people invest far too much emotional capital in what reviews "should be."

For me they are interesting reading material but I can't remember a purchase I've made that was based only on a magazine review, whether print or online.

Reviews are useful to me in the follow ways:

1. If I am in the market for something, reviews can be useful for adding candidates that warrant further investigation.

2. Reviews are useful for keeping up on industry gossip as respects what's new and fashionable. Those fads and fashions may be of interest to me or could be further evidence of how low we've sunk as a society. ;-)

3. Mainly I just enjoy reading about audio since it is an area of interest for me. Sometimes articles are educational and sometimes the author is full of it or comes from a direction that simply doesn't connect for me.

What magazine reviews will never be for me is some kind of holy writ or scientific reference text. I think people make a basic mistake when they try to elevate what is essentially hobby-based reading entertainment to a level of authority that is neither deserved or intrinsic to the format.

[Ajani]

I believe many people invest far too much emotional capital in what reviews "should be."

For me they are interesting reading material but I can't remember a purchase I've made that was based only on a magazine review, whether print or online.

Reviews are useful to me in the follow ways:

1. If I am in the market for something, reviews can be useful for adding candidates that warrant further investigation.

2. Reviews are useful for keeping up on industry gossip as respects what's new and fashionable. Those fads and fashions may be of interest to me or could be further evidence of how low we've sunk as a society. ;-)

3. Mainly I just enjoy reading about audio since it is an area of interest for me. Sometimes articles are educational and sometimes the author is full of it or comes from a direction that simply doesn't connect for me.

What magazine reviews will never be for me is some kind of holy writ or scientific reference text. I think people make a basic mistake when they try to elevate what is essentially hobby-based reading entertainment to a level of authority that is neither deserved or intrinsic to the format.

Very well said!

I read reviews for the same reasons... Nothing wrtten in review is Gospel.... It's entertaining and hopefully informative... but at the end of the day, it should never be taken as more than a basis for finding new products to audition...

[Feanor]

One thing I'm NOT looking for in reviews are lengthy rambles filled personal anecdotes and reflections, We get a great deal of this especially at TAS and Stereophile. Is this narcisistic or literary pretense? Either way, I don't find it entertaining much less useful.

[mijs]

Just informative, and I'd like to see more of them.

Comparos are done in motorcycle mags ALL the time and none of the manufactureres seem to suffer from them.

Bike A is slightly faster, but Bike B handles a little better, and Bike C is the most comfortable. Potential buyers can decide which ones they want to test ride.


Regardless of the type of audio equipment or the price range, a descriptive comparo between top contenders would be great. The reviewers can always find positive things to say about a quality piece.

What's the harm?

[Ajani]

Just informative, and I'd like to see more of them.

Comparos are done in motorcycle mags ALL the time and none of the manufactureres seem to suffer from them.

Bike A is slightly faster, but Bike B handles a little better, and Bike C is the most comfortable. Potential buyers can decide which ones they want to test ride.


Regardless of the type of audio equipment or the price range, a descriptive comparo between top contenders would be great. The reviewers can always find positive things to say about a quality piece.

What's the harm?

Absolutely nothing wrong with comparison... the issue is when they attempt to pick a clear winner... Since many persons assume that the winner of a group test of similarly priced products is the one they should buy... When in reality the one that came last may be the most suitable one for their musical tastes....

The issue with HiFi is that many of us want a reviewer to tell us what is best and what we should buy... No reviewer/mag can do that....

Audio Note AN-E might well be the best speaker ever made in the opinion of several experienced reviewers, but that doesn't guarantee that you will like it... Same thing for any product that a reviewer recommends or buys....

I think the biggest question may actually not be what do mags need to do to be more useful, but instead how to get consumers to understand the use of mags....

[Ajani]

One thing I'm NOT looking for in reviews is a lengthy rambles filled personal anecdotes and reflections, We get a great deal of this especially at TAS and Stereophile. Is this narcisistic or literary pretense? Either way, I don't find it entertaining much less useful.

lol sadly, that is true for many readers... I often skim through those sections of the review... As nice as it is to know what kind of music a reviewer listened to during the review, I don't care for paragraphs of detail on songs I've never heard and am not interested in hearing... But I suppose for person with the same musical tastes as the reviewer, such fluff may be interesting... same thing with the anecdotes and reflections....

[nightflier]

Well one thing I've suggested in another thread is to have only the top products from each company reviewed, this being the absolute best they can produce, so there is no excuse about whether the company skimped on anything to save for the next model up.

Personally I always gravitate towards the shootouts, maybe only because they are more entertaining to read. It irks me to no end when reviewers "compare" a product under review with something that hardly resembles it or with their "memory" of a past product they reviewed. The European mags at least are more fun to read, even if they don't have all the scientific data to back everything.

[manlystanley]

I also like to read foreign magazines for different perspective of reviews. For example, WhatHIFI goes ga-ga over the Proac speakers, but Norwegian and German magazines give them a big yawn. (Google translate is a big help with this.)

So, it make we wonder if Germanic peoples have a genetic predisposition to certain sound qualities.

Best Regards,
Stan

[mijs]

Absolutely nothing wrong with comparison... the issue is when they attempt to pick a clear winner... Since many persons assume that the winner of a group test of similarly priced products is the one they should buy... When in reality the one that came last may be the most suitable one for their musical tastes....

The issue with HiFi is that many of us want a reviewer to tell us what is best and what we should buy... No reviewer/mag can do that....

Audio Note AN-E might well be the best speaker ever made in the opinion of several experienced reviewers, but that doesn't guarantee that you will like it... Same thing for any product that a reviewer recommends or buys....

I agree. That's why reviewers shouldn't pick a winner. IMO, that's not their preogative. They should just report the character, build quality, and strengths/weaknesses of each piece compared to the others. Let the buyer decide what would work best for him. As long as the pieces tested are similarly priced and are respectable pieces to begin with, it will work.

[RGA]

So for example when testing some AN speakers John Atkinson didn't set them up in room corners as AN prescribes for best results. So what would the measurements have been if they were set up as specified? We don't know. Atkinson justified his approach on the basis that he test are speakers in a certain way to ensure fair and consistent testing -- that's objectivity, right??

Not really objective in the least - it is however easier but it forces a square peg into a round hole - his method doesn't do justice for a lot of speakers like, interestingly enough, panels. Though the problem with the AN E review has been discussed on the High efficiency forum of audioasylum by several engineers (and a speaker competitor named Duke). The problem I had with the measurement is that the manufacturer specs them in corners and Stereophile reviewed them in the middle of the room on axis at too high a height. Now even if you want to fit all speakers into the round hole you can't really claim it doesn't meet spec. The AN E in several other magazines and by a "more" technically knowledgeable guy in Martin Colloms (the guy JA goes to for help) measured them and got the numbers.

Peter Qvortrup noted that Stereophile does a disservice to panels and some other designs. Of course if you need the measurement to tell you it's good or not then you can shop via the Sears catalog because the ear isn't good enough to spend much more than that in my view.

[Ajani]

Wrong comparison. It is "objective" vs. "observational". To observe that a condition exists exists involves no subjectivity. Subjectivity occurs afterwards in either case with a judgment regarding that change. Two people can agree completely on a difference and yet arrive at different conclusions. I've known two prominent reviewers for decades and sometimes disagree with either as to the subjective side of what I find best - having heard exactly the same things.

That's a good clarification... I actually had to read your point twice for it to really sink in (hopefully I was just really tired when I first read it)...

Those of us who do remember that approach by Len Feldman and Julian Hirsch recall how utterly useless they were to convey any useful knowledge as opposed to information.

I never read their reviews, but I can imagine how worthless it would be to try and buy a stereo based on just the test measurements in Stereophile (for example)... As informative as measurements are, I can't see how they can be used in isolation...

In the comparative sense, sure. Remember too that reviewers (and store employees) can get any component at accommodation (I've purchased a couple myself that way), so it is a level field when comparing one to another.

I think TAS should return to its roots with the approach HP took long ago before he had to turn the reins over to new management.

rw

Certainly, comparatively between products of the same price... but between products of different prices, the comparison gets skewered...

The difference between a $3K amp and a $1K (at accommodation) in terms of percentages, is the same as the difference between a $9K and a $3K amp (at retail)... but while someone may be willing to accept the difference ($2K) in the first instance, the difference ($6K) in the second situation, for the same amps, might be totally unacceptable...

[Ajani]

Should they review mostly popular or exotic gear?

Major mags get criticized heavily for reviewing major brands just about every issue... Krell, Revel, B&W, KEF, NAD, Rotel, Arcam, Dynaudio, Musical Fidelity etc.... get constant reviews in The Absolute Sound, Stereophile, HiFi Choice, What Hifi? etc...

Some persons argue that these mags should focus more on smaller, exotic brands...

Interestingly, I find that the smaller mags/review sites often focus almost exclusively on smaller brands... sometimes even obsessing over small brands the way large mags obsess over large ones..

For example check the speaker reviews section of Dagogo and count how many reviews there are for Audio Note speakers... Just about every model of the AN-E Speaker has been reviewed there (seems almost every writer on the staff has reviewed and raved about an Audio Note speaker - not to mention the amount of AN DACs and Amps that have been reviewed on the site as well)....

So is obsessing over small, exotic brands any more useful to readers?

Would you rather read reviews of gear you might actually be able to audition and buy, or of gear you might only be able to get used (every few years) or purchase special order?

Should the goal be to have a mix of different gear, with a focus on popular brands, but still making regular space for the exotic?

What price range should be the focus of the reviews?

A constant criticism of some of the American mags has been the amount of attention that $100K speakers and equipment have received in recent years....

Should the focus be on affordable gear (let's define affordable as $2K or less per component for the sake of this discussion), Moderately Expensive $2K > $6K, Expensive $6K > $15K or Elite (>$15K)? Should there be a balance based on the amount of components in each category? Should the balance be based on what the average audiophile invests in his system?

[RGA]

Ajani

It depends on what is considered small. Audio Note is not a good example really because they're not small. They only do 5% of their business in North America. They have over 700 products and many of them are not even listed on the website. When I went to CES I asked peter why he doesn't make a Tuner. He said "we make tuners" - I said "then why is it not on the website?" he replied "then people would want them and we'd have to build them." They can't keep up with their order book as it is.

I understand the argument you're making that when we review stuff that is hard to find that it doesn't help the average audiophile living in smaller towns that don't have access to such gear - but it "should" be about the better gear out there not how many retail outlets carry it. All of the writers at Dagogo heard the stuff in the United States and I am in Canada. There are AN dealers in BC, Alberta and Ontario (though only Soundhounds carries enough stock to show it decently) - and even they can't get stock.

Reviews can actually do a disservice to some makers because good reviews can generate a lot of interest and possibly a lot of sales. But if you are small company and demand heavily outstrips supply or the ability to supply then companies tend to move into larger buildings and hire more people. This put Reference 3a out of business when they were just called 3a. They moved Daniel Dehay lost the ability to oversee large production and the quality suffered heavily putting them under. The second time around he made sure to do it right under the Reference 3a name.

The problem too is that reviewers don't know what is available in your town. PMC is pretty big here but may not be in your city. Paradigm is big in North American and practically non existent in Europe. Audio Note is very big in Russia, Poland, Sweden, Croatia, Vietnam, Denmark, Britain, etc.

I believe there are only 10 dealers in the U.S. but one of the biggest and best is Audiofederation who are also the distributors.

On to a more general note - review sites can be valuable to put stuff off the radar on the radar. Anyone can go and audition a B&W - big deal - why does anyone need a review to tell you what you heard for yourself. I see the review being able to tell you about a Trenner and Freidl loudspeaker company making a speaker that can stand with anything. Until CES I had never heard of them before even though they've been around quite awhile in Europe making highly regarded products.

Heck Sugden is hard enough for most people to find and audition and they've been around for over 40 years.

Reviewers are also in it partially for themselves. I want to spend months with stuff I like and would consider buying - I don't want to spend time with the B&W 705 which I think is obscenely overrated - just because every town is selling it.

I think though that Dagogo and other sites review mainstream stuff as well. Generally though I find the mainstream stuff is a significant step down across the board from the smaller makers who have passion and often cost no concern approaches over choosing a price point and then making something to fit it.

[nightflier]

RGA, the popularity malaise that you describe is exactly what Klaus at Odyssey told me when he was building my Candela and I complained to him about the website being out of date. He as only two other techs on staff and can't keep up with demand, not to mention keeping the website up to date (although he has found someone to do it now). He wishes that he had more time to grow the business side of Odyssey, but he's too busy building the gear. Outsourcing really isn't an option for him since that would increase costs and affect QC. He's actually retracted from previous outsourcing deals (The Nightingale and Circe speakers) exactly for those reasons.

To answer your other point, maybe there ought to be a magazine (or site) that doesn't claim to be a reviewing rag, but instead just showcases new gear with specs and pics. I know there are some mags that sorta do that, but I was thinking about one that would be a little more impartial and less advertising-driven. Also, if there is one thing that would make magazines like Stereophile more appealing is more photos, particularly of the back-sides of gear.

I also think that online sites are a useful source for reviews in general because it's a great place to aggregate many opinions together. Granted, this tends to favor the products with more distribution, but over time, I think the better products will eventually emerge. This site (Audio Review) is a good place, but I do wish it was more complete. It actually surprises me how few of us who contribute posts on the forums almost daily actually also write reviews. Maybe more incentives might help?

[E-Stat]

Audio Note is...

Every post is always about AN...

rw