Blind listening comparisons

[Brett A]

Brett nice of you to join us over here. Any word on AK? Was it a planned shutdown?

Dave (Grumpy) posted to Political Chat stating it was a hardware issue. They're working on it.

It's nice to be back over here. I haven't stopped in at AR for a while. It was these folks who got me back into vinyl for which i am very grateful.

[GregLee]

Do people other than scientists actually turn their listening rooms into test chambers?

I'd guess not so many, counting audio engineers as scientists. I think I profit by blind testing, though I've never done any myself, because my favorite speaker company, Axiom, is said to make extensive use of blind testing in their product development. There is lots of interesting detail, also, on Dr. Sean Olive's blog Audio Musings, of Harman International, from which one can gather that doing it the right way, the way the pros do it, takes a tremendous investment of equipment, expertise, and time+effort.

[Ajani]

I'd guess not so many, counting audio engineers as scientists. I think I profit by blind testing, though I've never done any myself, because my favorite speaker company, Axiom, is said to make extensive use of blind testing in their product development. There is lots of interesting detail, also, on Dr. Sean Olive's blog Audio Musings, of Harman International, from which one can gather that doing it the right way, the way the pros do it, takes a tremendous investment of equipment, expertise, and time+effort.

Interestingly enough I'm a big fan of manufacturers using blind testing in the design process (though like PoppaC, I can't see myself setting up my listening room to do it during an audition)... One of my 2 favourite brands, Revel, uses DBT extensively in their design process (as you mentioned re Harman International)...

I've never tried Axiom, but I've always been interested to hear how they sound, and their shipping rate to Jamaica is quite attractive...

[Geoffcin]

One of the reasons that Canadian speaker manufacturers have a leg up on a lot of the industry is that they have access to the Canadian National Research Council's testing lab. Think millions in state of the art testing equipment! There you can do a DBT and really get some science out of it. How anyone could do a DBT at home and come up viable data without such an investment when major speaker manufacturers turn to the experts is beyond me.

[RGA]

When the complete zippered hood is not available.

bring out the gimp!

Blindfolds are very helpful in many instances when you have not had enough beer.

[tube fan]

I am totaly against blind testing when it comes to audio. I have to listen to the equipment for a period of time before I decide if it is right for me. In quick blind comparisons I can hear differences but not what is best over the long haul. I think blind listening tests are bull****.

I agree, but only that longterm blind listening tests are needed. Of course, the same is true for wines. Of the last ten blind wine tastings I have attended, the most inexpensive wine won 4 times (prices from $10 to 250). I suspect that the same thing would occur in blind listening tests, both short and long durations. Yes, I bet most "golden ear" experts would not rate their favorite, very expensive components at the top. I bet many cheaper, especially tube driven systems, would be preferred over the unbelievably expensive systems, especially digital and ss based ones.

[E-Stat]

How could there be such a war? Do you think those doing DBTs don't use ears, or are using something other than ears? What does "ears only" mean to you?

The biggest problem with most double blind tests is they involve a series of unproven assumptions. Most require switch boxes which are assumed to be totally transparent based upon simplistic metrics that are themselves *blind* to inherent faults. Long ago, Frank Van Alstine pointed out that comparing amplifiers with ABX boxes was completely flawed because they required common grounds to prevent horrible switching transients. In practice you are now comparing A+B to A+B. Theory works great when supported by empirical evidence.

I have no problem with blind testing per se, although it tends to be of the single blind category devoid of adding superfluous boxes which always mix and match electrical characteristics of the two devices under test. The only DBTs I trust involve computer based source material which are devoid of obvious electrical faults.

rw

[poppachubby]

I agree, but only that longterm blind listening tests are needed. Of course, the same is true for wines. Of the last ten blind wine tastings I have attended, the most inexpensive wine won 4 times (prices from $10 to 250). I suspect that the same thing would occur in blind listening tests, both short and long durations. Yes, I bet most "golden ear" experts would not rate their favorite, very expensive components at the top. I bet many cheaper, especially tube driven systems, would be preferred over the unbelievably expensive systems, especially digital and ss based ones.

tube fan, I like you but I will play devil's advocate for a second.

I have heard you say on more than one occasion that you have blindly picked out your favorite (AR?) speakers from a line up of others.

If you are able to do that, then why can't a "golden ear" (and they do exist) pick out a fine piece of tube kit from horrid state?

[E-Stat]

There is lots of interesting detail, also, on Dr. Sean Olive's blog Audio Musings, of Harman International, from which one can gather that doing it the right way, the way the pros do it, takes a tremendous investment of equipment, expertise, and time+effort.

Dr. Olive also makes erroneous assumptions which invalidate the data if you really care about sound reproduction. Their tests cater to the lowest common denominator of listener and he thinks that speaker placement is irrelevant to performance. I'm amazed at such ignorance. I've had a discussion with him where he is apparently oblivious to the question of optimum placement - which the Harman "shuffler" is incapable of accommodating. Once again. what looks good on paper doesn't necessarily translate to the more complicated environment of the real world. If you really don't care about system optimization, then his simplistic trials may have some bearing.

Sean Olive

He never responded to my observations about the value of speaker placement. I guess if you assume that such makes no difference to performance, then you will make simplistic assumptions like his. I summarily ignore those who don't understand simple concepts.

rw

[GregLee]

Dr. Olive also makes erroneous assumptions which invalidate the data if you really care about sound reproduction. Their tests cater to the lowest common denominator of listener and he thinks that speaker placement is irrelevant to performance.

Does he really think that placement is irrelevant? It's hard for me to believe. Maybe you could give us a reference to substantiate this? I read the link you gave, and there seems to be no relevance to what you've said here.

[RGA]

So DBTers don't trust their ears? If you're not blindfolded, you trust your ears only, but if you are blindfolded, you trust something other than your ears? Oh, wait, I get it --- you're trying to imply that the DBT crowd are using their minds, as well as their ears.

I think that the arguments get out of control on the issue. A DBT is a tool that is used in many different fields and was originally used in the medical profession where patients would get either a placaebo or the actual drug. The data is collected and it's either A or B. The trouble with the Audio Versions of the test and to a degree the educational uses of these tests is the fact that unlike the medical DBT - there is a "testing" factor where someone has to make a choice. No amount of softening can take the "test stress" factor out of the equation. The mere fact that the subject knows they are being "tested" and has to make a choice is a factor that diminishes the "validity" of the DBT in audio.

There are many bright individuals who can write you an essay on a subject but on a test they can freeze up. This is in part why educators hate standardized tests (at least partly) because in fact some people are terrific test takers but not all that bright - they wind up with A's while numerous brainiacs can perform horribly under the test environment. Of course there are also brainiacs that do well on both and some people who do poorly on tests because they're looking up to Forest Gump (or Dubya Bush:-)

As most people are aware the brain is divided into two hemispheres generally refered to as the intellectual logical mathematical side while the other is the artistic music side of the brain. In a regular listening session it is the latter side of the brain that is engaged. In a test invironment (and they have actually hooked up people's brains and tested them), the emotional artistic side is not activiated in the same way. The brain is now in a different mode of listening - not so much for relaxation but concentrating and using the analytical side for identification, processing information, and solving a problem.

It is therefore very ligitimate to argue that the test is no longer truly recreating the normal way a person listens to music or even how they make a buying decision.

All of this does not mean you just chuck out blind listening but the T in the DBT is problematic and many fields including education are moving away from them.

Why is a blind listening important - all the obvious reasons:
Price - if it is more expensive it will be assumed to be better

Looks: mostly males buy audio equipment - males are seduced by visuals and it serves evolution well as we hunt for the most fertile healthiest looking women (somewhat subveted by advertising over the years but still...)

Name brand recognition - more advertising more talk - generates a "me too" attitude as we want to follow the most powerful monkey (err star in humans case - which is why we have movie stars advertising for cologne - now if audio companies did the same you may see more sales for a given company - B&W kind of does that with Skywalker sound and George Lucas)

Build quality/weight - if it is heavy looks beautiful costs a fair bit then it must be better. Consider McIntosh amplifiers - they're built like a Tank - have soothing Blue meters, have great name brand appeal and is a kind of "me too" product. I think most people would prefer to turn on a mcIntosh in their system than staring at a boring black box

Unique - this is opposite of the me too approach where one wants to stand apart from everyone on their block - it may be better or it may serve as a conversation piece - "that's a speaker - it looks like a giant scratching post?" (panel speakers) or that's an amplifier? - what are those little lightbulbs (Tube and SET amplifiers). This doesn't mean they're not bought on sonic preference but illustrating the bias that could be present (I own a tube amplifier).

Specifications/measurements More power is viewed as better. It's simpler for manufacturers to sell their gear this way. 50 watts for the $300 model 70 watts for the $600 model and 200 watts for the $2000 model. Build a product that has flat measurements into the popular measuring standards of the current decade and you can sell that as being accurate and with it being approved as accurate it generates a bias in people. Manufacturers are not stupid - especially big companies that hire big marketing departments - create in house tests, white papers, hire some Phd's and now your packaging up "science" to sell to the folks who will trust the manufacturers' version of things. Kind of like trusting the drug companies, corporations, and the food industry - all of which have their big hands in the Universities. Regardless Athiests make up the biggest minority group in the United States at about 20% and generally science will appeal to us. So why not package up measurements with blind tests and we'll go for it over any other way of attempting to convince us. Granted I am Canadian so I have a healthy distrust of science being conducted from a company that is selling me their product which naturally always wins their tests against competitors who had no say in the test or were able to review the set-up or had any say whatsoever.

Hi-Fi Choice is the only magazine I know of that does in fact conduct blind listening. They have a set of speakers in the room amp cd player etc in the room. They may test 7-10 CD players or what have you. All the CD players are level matched. They have a panel of listeners - maybe half dozen to a dozen people. All of the CD players are hiddend from view. So none of them know what is playing. In some cases manufacturers of the products under test comprise of the listening panel. What is interesting is that sometimes the manufacturer doesn't even choose his own unit as being the best in the listening session. The panelists take notes as they listen.

So you have as close as you're going to get to a DBT from the audio press. The components are all level matched, there is more than one person listening - a panel of listeners, and no one knows what units are being played. So it removes all of the "important biases" and it also doesn't have a stress component to the listening because the listeners are not being tested.

Hi-Fi Choice does not do all their listening in this manner - generally it is done for their shootout. Also, they occassionally have a sighted session tie breaker where the editor will tell you the results and then describe how he might disagree with some of the findings. I have less problem with this in fact because they generally tell the reader all of this so you can decide for yourself to put more weight into the panel session or more weight into the reason perhaps to not follow it.

John Atkinson put stock into them for awhile and sold his pricey gear for relatively cheap gear but after the short duration DBT session wore off he felt that the long term subtlties warranted having his better pricier gear. Then again - who knows. Maybe that was a tactical thing to show the readers that you need to spend large. Otherewise why would you need a Stereophile - just buy a $199 amp at Sears and matching $39 CD player and be done with it. Without expensive gear - there is no point of high end magazines. JA would have shot himself in the foot if he came out for $1k and under amplifiers or insinuated that all properly working SS amplifiers sounded the same. I happen to believe JA does hear the differences - just playing devil's advocate here.

[E-Stat]

Does he really think that placement is irrelevant? It's hard for me to believe. Maybe you could give us a reference to substantiate this? I read the link you gave, and there seems to be no relevance to what you've said here.

Ok, I guess I need to help you read his comments.

"I'm not sure what you mean that the speaker shuffler, listening room, amps and cables have to be adjusted to compensate for a particular type of speaker such as dipole. Wouldn't that be a rather biased test if each loudspeaker required a unique cable, position and amplifier to sound good?"

Yes Sean, one does have to compensate for the type of speaker when considering optimum speaker placement. Would it be a biased test to compare speakers in their optimum positions? Huh? What are you smoking?

rw

[RGA]

I agree, but only that longterm blind listening tests are needed. Of course, the same is true for wines. Of the last ten blind wine tastings I have attended, the most inexpensive wine won 4 times (prices from $10 to 250). I suspect that the same thing would occur in blind listening tests, both short and long durations. Yes, I bet most "golden ear" experts would not rate their favorite, very expensive components at the top. I bet many cheaper, especially tube driven systems, would be preferred over the unbelievably expensive systems, especially digital and ss based ones.

Martin Colloms of Stereophile already conducted such a test where an old no feedback tube amp beat expensive SS amps that were the best out at the times. But get this - the blind test was done with all the TOP SS designers and they chose the tube amp as sounding better than their own amps. You don't have to BET that the experts prefer no feedback - they do. here's the link http://www.stereophile.com/content/f...thout-feedback

[tube fan]

This line however I do take issue with. I think that there's a very strong correlation in the lower price points. When you move up from $20 speakers to $100 speakers, there is definitely a marked improvement. And likewise when you move from $100 to $500.

I agree though that there is a point of diminishing returns, and that's where you get into the stickiness of whether any perceived improvement in performance is worth the often exorbitant price increases.

I am NOT talking about $20 speakers. I AM talking about under $10,000 speakers compared to $150,000 ones. I AM talking about something like my system: AR SP8, AR D-70, Fosgate Phono, VPI Scoutmaster, Benz Ruby 3, and Fulton J or Dunlavy SC-IV speakers compared to ones costing $400,000 (like ones I heard at the California Audio Show).

Bias is an all too human trait. When we KNOW we are tasting a Screaming Eagle $1,500 bottle of wine, we tend to look for positives. When we are comparing a Benzinger $15 bottle to the Screaming Eagle BLIND, the resulting evaluation will often be quite different. I recall a series of wine tastings conducted by the NYT. All were Pinot Noir based, from France, Oregon, and California. Tasted and rated knowing the bottle name, the French wines were rated first, then Oregon's, followed by California's. Tasting EXACTLY the same wines blind, California wines came in first, followed by Oregon's and French wines. I've seen it hundreds of times in blind tastings. I bet a similar thing would occur in blind listening tests, short or longterm.

[tube fan]

tube fan, I like you but I will play devil's advocate for a second.

I have heard you say on more than one occasion that you have blindly picked out your favorite (AR?) speakers from a line up of others.

If you are able to do that, then why can't a "golden ear" (and they do exist) pick out a fine piece of tube kit from horrid state?

Poppa, I am NOT claiming that there are not significant differences in equipment, especially between ss and tube, and analogue and digital. I am NOT contending that no one can consistently rate and identify equipment. I just happen to be quite good at consistently rating and identifying wines
tasted blind. I AM contending that many "golden ears" could not rate consistently their favorite expensive equipment vs more modest ones.

[Ajani]

Dr. Olive also makes erroneous assumptions which invalidate the data if you really care about sound reproduction. Their tests cater to the lowest common denominator of listener and he thinks that speaker placement is irrelevant to performance. I'm amazed at such ignorance. I've had a discussion with him where he is apparently oblivious to the question of optimum placement - which the Harman "shuffler" is incapable of accommodating. Once again. what looks good on paper doesn't necessarily translate to the more complicated environment of the real world. If you really don't care about system optimization, then his simplistic trials may have some bearing.

Sean Olive

He never responded to my observations about the value of speaker placement. I guess if you assume that such makes no difference to performance, then you will make simplistic assumptions like his. I summarily ignore those who don't understand simple concepts.

rw

What makes his failure to acknowledge the effect of speaker placement truly puzzling is the fact that Revel's Performa and Ultima Lines both have controls on the speakers, for adjusting treble and bass, to account for being placed near to room boundaries or in open space... So clearly he must understand that placement has some kind of effect on a speaker's performance....

[poppachubby]

Poppa, I am NOT claiming that there are not significant differences in equipment, especially between ss and tube, and analogue and digital. I am NOT contending that no one can consistently rate and identify equipment. I just happen to be quite good at consistently rating and identifying wines
tasted blind. I AM contending that many "golden ears" could not rate consistently their favorite expensive equipment vs more modest ones.

Don't get defensive, I'm not trying to make you look foolish.

I don't see how your reply answers my question. What you are talking about is apples and oranges. If someone can't pick out expensive gear vs. inexpensive, how are you able to pick out a pair of speakers vs. others?

My point is not expensive vs inexpensive, simply A vs B. I think some people probably can pick out a piece of gear from others, just as you can pick out the AR speakers.

[02audionoob]

I am NOT talking about $20 speakers. I AM talking about under $10,000 speakers compared to $150,000 ones. I AM talking about something like my system: AR SP8, AR D-70, Fosgate Phono, VPI Scoutmaster, Benz Ruby 3, and Fulton J or Dunlavy SC-IV speakers compared to ones costing $400,000 (like ones I heard at the California Audio Show).

Anyone would have a line where they no longer perceive a difference or claim that no difference could be perceived...and the line would often be right about at their own system. I would bet there are people in this hobby who would claim their own $3,000 can perform up to the level of $10,000 speakers. And then there would be people who claim those $3,000 speakers can't be distinguished from their own $1,000 speakers. We all make our own decisions about this stuff and we quite often base our comments on the subject on our own choices.

[GregLee]

What makes his failure to acknowledge the effect of speaker placement truly puzzling ...

The whole point of the elaborate method of switching speakers used in Olive's test room is to get the speakers into exactly the same position, because speaker placement does obviously play a large role. To argue that because Dr. Olive won't make ad hoc adjustments in placement for speakers under test he must not understand the importance of speaker placement just makes no sense at all. To make a fair comparison, speakers must be in the same place.

[Ajani]

Anyone would have a line where they no longer perceive a difference or claim that no difference could be perceived...and the line would often be right about at their own system. I would bet there are people in this hobby who would claim their own $3,000 can perform up to the level of $10,000 speakers. And then there would be people who claim those $3,000 speakers can't be distinguished from their own $1,000 speakers. We all make our own decisions about this stuff and we quite often base our comments on the subject on our own choices.

I made that point on diminishing returns in another thread a short while ago... IMO, diminishing returns in HiFi is virtually worthless as it is 100% subjective... Further, it seems to just come down to the question of how much someone is willing to spend on gear...

Joe A will say that he gets 95% of high end performance with his $1K amp.
Joe B will claim 95% is achieved at his $5K, and $1K amps are midfi (whatever that means).
Joe C will claim 95% with his $25K monoblocks and claim that it sounds better than $250K monos (which he believes are for persons who just want to spend money, blah blah blah)...

However check back on any of these Joes 5 years later and you maybe 'surprised' to find they've moved up the chain to a much higher priced amp, and are now claiming that it lifts a veil off the music and presents the musical information with new dimensionality, emotional complexity and transient existential harmonality or whatever...

IMO, the only thing you should focus on is getting the system that gives you the most satisfaction for your budget... As nice as it is to feed my ego, by claiming my system sounds as good as, or better than, setups costing 5 - 10 times as much, it is just meaningless boasting....

[Ajani]

The whole point of the elaborate method of switching speakers used in Olive's test room is to get the speakers into exactly the same position, because speaker placement does obviously play a large role. To argue that because Dr. Olive won't make ad hoc adjustments in placement for speakers under test he must not understand the importance of speaker placement just makes no sense at all. To make a fair comparison, speakers must be in the same place.

I think my full post makes it clear that Dr Olive must understand placement...

Now the question to ask is whether it is really fair to have all speakers in the same place...

Consider speakers designed to be used in a corner like from Audio Note. Is it fair to pull them out 5 feet from the front and side walls, for a comparison with a monster tower that needs space not to sound boomy? At 5 feet out the Corner speakers will have weak bass response while the monster towers will have ideal bass... If we switched positions and placed both speakers in the corner for the test, then the Corner speakers would have good bass, while the monsters bass would sound bloated and overbearing...

So there is a valid question of whether shuffling all speakers into the same position is actually a good test... It should work fine for most traditional cone speakers (which are designed to work in free space), but it would not be fair to corner loaded speakers and probably dipoles (as E-Stat said)...

[E-Stat]

It should work fine for most traditional cone speakers (which are designed to work in free space), but it would not be fair to corner loaded speakers and probably dipoles (as E-Stat said)...

Which was the context of the discussion. If you read some of his other blogs, you find a thinly veiled amazement why some folks like Martin-Logans in particular when the Harman tests with them don't register as well. Placing dipoles in exactly the same position as a monopole will never optimum for one or both of the speakers. I get the smoothest measured low end response with my full range stats eight feet out into the room. Putting them on the "shuffler" and considering that a "fair" comparison indicates a lack of understanding about dipolar response. Not to mention an indifference to the amplifier used for driving reactive loads. His biases are quite clear.

edit: Here's a part of the abstract to one of his AES papers: "The experimental results from both tests show that listener preference ratings for different loudspeakers are significantly influenced by the loudspeaker location within the room. In fact, the positional effects can be larger than the subjective differences between the loudspeakers themselves.

No fooling.

rw

[Brett A]

Originally Posted by poppachubby
Do people other than scientists actually turn their listening rooms into test chambers?

I'd guess not so many, counting audio engineers as scientists. .

I'd like to see evidence of these actual scientific studies. So far, I have not.

When I think of people doing DBTs and ABX-ing audio. Scenes like the one below come to mind; a group of guys getting together to find something out for themselves and then writing about it on the interweb.

So far, I have yet to see an objective, scientific assessment of the appropriateness of using DBTs for audio. Can anyone offer links, please?

When the scientific community (at least certain types of current of neuroscience research) wants to runs tests that rely on perception, they often have to do years--and I mean years --of testing to arrive at what the legitimate final test will be.

So far, I have not seen any scientific scrutiny of DBTs for audio testing. Only the occasional EE and/or hobbyist saying they are "proof" of something from a scientific stand point.

To me, all they prove is that statistically, a majority of people cannot discern some differences that they might think they heard if they had visual input, and some can. Which is different than saying there is no difference.

[Ajani]

Which was the context of the discussion. If you read some of his other blogs, you find a thinly veiled amazement why some folks like Martin-Logans in particular when the Harman tests with them don't register as well. Placing dipoles in exactly the same position as a monopole will never optimum for one or both of the speakers. I get the smoothest measured low end response with my full range stats eight feet out into the room. Putting them on the "shuffler" and considering that a "fair" comparison indicates a lack of understanding about dipolar response. Not to mention an indifference to the amplifier used for driving reactive loads. His biases are quite clear.

edit: Here's a part of the abstract to one of his AES papers: "The experimental results from both tests show that listener preference ratings for different loudspeakers are significantly influenced by the loudspeaker location within the room. In fact, the positional effects can be larger than the subjective differences between the loudspeakers themselves.

No fooling.

rw

That's part of the challenge with scientific testing... There are just so many variables to factor in... So once you start to simplify a model too much, you severely limit its usefulness...

I have great respect for the work done at Harman, especially since I am fond of the Revel sound... But I recognize that their testing cannot be applied universally to all types of speakers and amps...

This discussion reminds me of a thread I read, this morning, on another forum: the OP had just played around with his setup for the hell of it - he moved his speakers from the long wall of his room to the short wall (20x13 room) and was utterly shocked at how much better his setup sounded...

[GregLee]

I'd like to see evidence of these actual scientific studies. So far, I have not.

There are some brief characterizations of such studies along with references in the Olive blog which I already provided a link to.