We need an "Audiophile Shootout"

[cjpremierfour]

I read about audio with music in the background, but I spend more time reading about audio than listening to music. That's the problem! It seems that this hobby is full of unknowns and judgements by others that make purchasing a difficult decision.

I think that this could be corrected by a yearly meeting where budget products will call out higher end gear and compete. There needs to be a Demo show where "Audiophiles" could cast their vote of approval. Rules should be that a "Product" should cost as least double or higher.

Example:
Emotiva XDA-1 DAC vs any $500 dac
Pioneer SP BS41 speaker vs $400 speaker
Technics SL1200 turntable vs Rega / Linn / VPI
DIY Pass Gain Clone vs Pass Aleph 0s
Radio Shack - RCA / Speakers cables vs any Hi End cable
Cheap Capacitors vs Hi End Capacitors
PSB Image T6 vs Focal Chorus 826w
Etc.......
This event should feature nothing but black curtains ( no peeking ) in ever room. Nothing but choice A , B or no difference.

I'm not sure who started the thread here, but it was something like the "Yearly State of Audio". That would be ideal to release the results to the audiophile community.

[02audionoob]

I read quite a bit about audio, too. One of the drawbacks to all my reading, though, is all the negativity. I'd rather just let the high end be what it is, without so much questioning of price points. I don't expect every $4,000 pair of speakers to be equal and I don't expect them all to be twice as good as a $2,000 pair. In this world you ask for what you think you can get for a product. It isn't necessarily going to be proportionate.

At some point, in my audio decisions I decided to just get something. There's great-sounding audio gear everywhere. About the only way I figured I could go wrong is to believe everything I read.

[Poultrygeist]

The sad thing is the great majority of readers here will never have a chance to hear the gear I love so much.

There are no opportunities to hear crossover-less bi-amped open baffles, single driver full range speakers and SET amps other than at someone's home. At audio fests they only showcase high end commercial offerings which have little in common with my passion.

Robert Frost sums up my audio journey best in the following quote: "Two roads diverged in a wood and I took the one less traveled and that has made all the difference"

[TheHills44060]

...It seems that this hobby is full of unknowns and judgements by others that make purchasing a difficult decision.
Your own 2 ears should be making the decision. Simple as that. Why would you let a review play any part in your conclusion about the way something sounds to you?

[RGA]

Hi Fi Choice and What Hi-Fi do shootouts. The former does them in level matched blind conditions and have a panel of people evaluating from reviewers to manufacturers themselves. They wind up choosing a winner out of the group they test.

Black gates were compared

Audio Note (http://www.audionote.co.uk/articles/art_black_gate_story.shtml)

And This article has speaks for itself - they compare 21 capicitors at many price points from the very cheap to the expensive - they list them in order from worst to best.

http://www.vhaudio.com/21capacitorshootout.pdf

Evaluating the individual parts does make sense since all amplifiers are a series of parts - garbage parts = garbage sound no matter what fancy box they put the parts in.

[Feanor]

Hi Fi Choice and What Hi-Fi do shootouts. The former does them in level matched blind conditions and have a panel of people evaluating from reviewers to manufacturers themselves. They wind up choosing a winner out of the group they test.
...
Bear in mind that that Hi-Fi Choice's blind testing is by no means scientifically rigorous "double blind testing" (DBT). While I like their concept a whole lot better than sighted testing, given the very large number of components they review I still have a lot of doubt about the validity of their results.

Further, their testing is for preference based on mainly subjective criteria, and there are not objective or verifiable descriptions of what the reviewers' personal preferences are. (Personal aside: I think the preferences of many reviews today is for a richer, chewier sound than is really accurate -- love those Mars Bars.)

[Poultrygeist]

Since hearing how much difference the $50 Obbligato's made in my phono stage I'm interested in cap reviews. For the money I see folks investing in IC's they could just as easily upgrade their caps with far better results.

[Mash]

It is a fool's quest to simply compare one speaker to another, one CD player to another....

Our audio memories are rather short in these situations because we do not really "know" the pieces of audio gear we are trying to evaluate. So... what to do?

Attend live recitals or small group performances that feature the music genres you prefer. The intimacy and the limited number of instruments will help you focus on the specific instruments and learn their sounds. Then use similar music to evaluate whatever equipment you are interested in, because your live music memory will be quite long.

[cjpremierfour]

It is a fool's quest to simply compare one speaker to another, one CD player to another....

Our audio memories are rather short in these situations because we do not really "know" the pieces of audio gear we are trying to evaluate. So... what to do?

Attend live recitals or small group performances that feature the music genres you prefer. The intimacy and the limited number of instruments will help you focus on the specific instruments and learn their sounds. Then use similar music to evaluate whatever equipment you are interested in, because your live music memory will be quite long.

This "Audiophile Shootout" idea came to me after reading budget equipment reviews. I was looking into turntables and DACs where review after review would compare it against a higher cost model. Therefore, the cost of a product becomes the reference for quality and / or sound.

This $250 DAC sounds as good as any $500 DAC. What does that actually mean?
What does a $500 DAC sound like?
What does a $1000 MC cartridge? etc.

What I'm thinking, we Audiophiles need a score. In sports, we have winners and loser and I would like to see something like that in this hobby. Put 100 audiophiles in a room and put up a number! We need results, not adjectives that reference it to a higher cost model.
Put the products on a floor, with all things being equal and let a group of Audiophiles make the call.

[RGA]

All of the review methods and DBTs have serious problems on numerous fronts. The biggest problem with the DBT is the T as it is a proven fact that placed in a test stress situation no matter how benign it may seem it diverts the brains attention away from that which you are actually testing ie; listening to music is not the same as taking a test and once blended it's not the same experience anymore. Hi-Fi Choice IMCO has taken the test out of the equation - kept the most important aspects (removing all forms of sight bias, price bias, name recognition bias) and level matched them (defeating the Munson effect that louder is deemed better).

The problem is of course as Feanor noted that you don't get to see every individual's response - so it's more of the most people who like it wins. But that is absolutely no different than what Harman puts out there - combined scores - or for that matter the abysmal science from the Oakland University site with the ABX DBT comparators that combine everyone's scores to the .05 statistical significance level.


Then we have the most common review style - bring in a CD player into the reviewer's system - he compares it to other CD players that came through. There is the issue of memory but detailed notes and experience means memory isn't all that critical since that is the point of taking detailed notes. The real problem is the time frame - products come and go so fast that it's far harder to compare current models to each other.

The measurements are not all that useful because they're all woefully lacking - and don't necessarily coincide with what the reviewer is choosing to buy. And no we can't conclude they're choosing less accuracy because the human ear brain is far more sophisticated than any audio measuring system. We know what sounds more like a symphony and what sounds more like a piano and more like a person talking to us or singing to us in a given space.

A/B blind level matched sessions can be done very easily - without a test add on and is completely valid.

It's so easy.

Step 1 - two identically dimensioned rooms (hotel whatever) are taken (that two companies agree would be deemed the "average" listening space of the "average home" living room - CES/The Show for example is not bad since the room is constructed from the same materials that 95% of homes are made from and the space is certainly the same as most living rooms I have seen in 30+ years. Indeed, better than most with carpets and furnishings as good or better than most "average" homes.

Company one and two sends their chief engineer to set up the speakers in the room so it is as good as it can be. (just like CES since they get the blue prints months in advance and know what the room shape is - if they're lazy oh well).

So you have to competing speakers and two identical rooms. You give them a a week to use whatever room treatments they deem necessary and that the average home user would likely employ. (or as much as they like). The speaker makers choose whatever front end gear they wish to use. Top of the line "Accuracy-R-Us" soilid state amplifiers they choose or SETs - up to them.

Black light is used to hide all of the equipment - no one sees it. The volume level is matched in both rooms at the listening chair.

A mixed CD is played - both rooms have the exact same music in the exact same order at the exact same volume level.

Each listener has a card (one yellow and one blue). They can listen to both rooms as often as they want. They drop the yellow card in the box that is sitting outside the room they liked the best.

They then have a second pass through - this time they get to control the volume - louder softer whatever - they do that for both rooms (because it's more like being a home user which a DBT fails to do). They then place the blue card in the box of the room that they prefer.

The CD has a sampling of well recorded Classical and Jazz pieces. All listeners will be classically/jazz professionally trained musicians and composers. There will be at least 50 listeners.

Part 2 is exactly the same but the music sampling will be rock/pop/house etc. These listeners will all be working musicians (trained or not) who work playing this music (DJ's included).

It's a test in the sense that one has to make a choice - but the choice is simple - does room A or B sound more like the real thing - if classically trained musicians can't tell - then ...

[RGA]

What I'm thinking, we Audiophiles need a score. In sports, we have winners and loser and I would like to see something like that in this hobby. Put 100 audiophiles in a room and put up a number! We need results, not adjectives that reference it to a higher cost model.
Put the products on a floor, with all things being equal and let a group of Audiophiles make the call.

The problem though is so what? Let's say 100 people listen to Product A and product B. Let's say 98 people like product B better.

Only 2 people liked product A better.

As a shopper you might play the odds and choose B because "most people" like it better. But most people might be clueless tone deaf dimwits. What if the 2 are Itzhak Stern and John Williams and the 98 are Keith Richards types only more stoned?

Even in my mini-article above - even with all the level matched blind testing hoopla - if it's not 100% it's not telling you anything.

There is always going to be someone who likes a Big Mac over Gordon Ramsey burger, or SS over SET. A lack of good taste is abound :biggrin5:

What most people believe doesn't make it so - ahem religion.

[RGA]

One final thought:

Everyone wants to gain the high ground - evolutionary tactic to be the top monkey.

Consider that everyone comes at the audiophile from varying perspectives:

New Technology types - new must be better than old after all why would they replace the old if the new were not better.

(Think about it! now you make the counter argument)

Old technology types - why fix it if it ain't broke - that new stuff is just a gimmick

(Think about it! now you make the counter argument)

The objective types (look at the graph - it has a flat response good phase, low distortion) therefore it is more accurate

(Think about it! now you make the counter argument)

The subjective types - yes but this clearly sounds better so if it sounds better and it measures bad something has to be wrong with the measurements (or lacking in depth of measurements)

(Think about it! now you make the counter argument)

The snob types - (this speaker costs $25,000; therefore it must be better - and it has a big name)

(Think about it! now you make the counter argument)

The anything I can afford is the best and anyone who spends more was sold a bill of goods types: Spending more than $3k on a speaker is idiocy because according to a large company that sells these speakers and according to their white paper from their hired engineers and their non-independent testing it clearly states that nothing is better.

(Think about it! now you make the counter argument)

Then do the same thing for every other debate; CD VS LP, Panels versus boxes, SS/digital versus Tube, Cables, headphones versus speaker, caps versus caps, AC versus DC, and on it goes.

The capacitor shootout was interesting at the end - he noted that two different sounding caps were both the only two he would use for building an amplifier - but suggested that one appealed to the brain and one to the heart.

Music and literature and other arts do the same things - some of it is "beautiful" and some of it is "intellectual." What moves one person to tears bores someone else to death. Aside from the equipment just listen to the music - you either like a given artist's music or you don't - 10 people can listen to the same artist in the same mood in the same room and you can have completely different reactions to the music based on your background. Marketing companies are well aware of this when they pipe it through shopping malls and stores. What will get the to rush a purchase - what will get them to stay just a bit longer?

The equipment it is played back on is inaccurate - all of it. So once you accept that fact it is which one sounds the most realistic - you would think that that would mean it's accurate but perhaps not. Which unit plays to the brain or the heart and doesn't ignore the other.

I want an emotionally involving experience from music i listen to - the system that can generate a tear - the one that can have you bouncing to the tune or toe tapping. If it can't do this then it doesn't matter if some review or some graph or some expert tells you it's more accurate if you'd rather listen when it's in the off position then it's a pointless endeavor.

Personally I want it to be accurate "enough" to hide very obvious faults and then I want the EQ (Emotional Quotient) to be as high as possible. Then it's about a personal sliding scale as to how much of each you desire.

I recently posted that the ATC SCM 150 is in my top 5. Another speaker is the Audio Note E speaker - actually some similar design goals in them.

The ATC is more about the intellectual - The AN E has more heart. The ATC pulls apart the sound and in a recording studio makes a lot of sense - the AN E I felt is easier to simply sit back and relax to. Neither is really right or wrong it depends what you're after. What they both do well is not mangle the other - the ATC still has emotion and the AN E still has the intellectual - one just tilts one way a little more than the other.

The owner of the long time audio shop Soundhounds refers to two types of high end systems.

One type is the sit forward in the chair listening experience. The people who sit and are actively engaged in the hearing of the "sound" the system is making. (typically the audiophiles who talk a lot about soundstaging)

The other type tends to be the sit back in the chair and listen to the music without really noticing or caring what the stereo is doing. (typically the people who talk a lot about organic sound, sound of instruments)

He carries speakers and systems that do a good job catering to both kinds of people. The bad thing about being reviewer is we need to be forced to pull apart systems that do the second very well in terms that cater to the first system's priorities.

[Feanor]

This "Audiophile Shootout" idea came to me after reading budget equipment reviews. I was looking into turntables and DACs where review after review would compare it against a higher cost model. Therefore, the cost of a product becomes the reference for quality and / or sound.

This $250 DAC sounds as good as any $500 DAC. What does that actually mean?
What does a $500 DAC sound like?
What does a $1000 MC cartridge? etc.

What I'm thinking, we Audiophiles need a score. In sports, we have winners and loser and I would like to see something like that in this hobby. Put 100 audiophiles in a room and put up a number! We need results, not adjectives that reference it to a higher cost model.
Put the products on a floor, with all things being equal and let a group of Audiophiles make the call.
My current cheap DAC is, based on several DACs and CDPs I've owned is quite astonishing in its performance. To my, ear the cleanest, most transparent, detailed & airy device I've heard. This thing costs US$60!!

http://imgs.inkfrog.com/pix/bluericewin/x19GB2.JPG

The thing is, as a cheap, generic Chinese DAC, when is it going to get a review? Maybe at some obscure online site. What chance that it will be in shoot-out with brand name DACs costing even 6x as much, ($360), much less those costing more? Virtually none, I'd guess.

[Poultrygeist]

I enjoy the "Cheap and Cheerful" forum at AC as that's one place you can learn about cheap obscure Chinese gear.

Folks who won't stoop to the level of "Chinese dime store gear" are missing out on lots of fun.

[Feanor]

I enjoy the "Cheap and Cheerful" forum at AC as that's one place you can learn about cheap obscure Chinese gear.

Folks who won't stoop to the level of "Chinese dime store gear" are missing out on lots of fun.
Ditto that. :yesnod:

[Hyfi]

Hey, is a Polk Monitor 30 Series II at 89db efficient enough for a flea watt amp?

I'm getting my Hafler 945 repaired and may sell the set of 945 and 9180 amp and use the money for some fun for my gym setup.

[cjpremierfour]

My current cheap DAC is, based on several DACs and CDPs I've owned is quite astonishing its performance. To my, ear the cleanest, most transparent & airy device I've heard. This thing costs US$60!!

http://imgs.inkfrog.com/pix/bluericewin/x19GB2.JPG

The thing is, as a cheap, generic Chinese DAC, when is it going to get a review? Maybe at some obscure online site. What chance that it will be in shoot-out with brand name DACs costing even 6x as much, ($360), much less those costing more? Virtually none, I'd guess.

Feanor just made my point! The "Shootout" would require a manufacturer of a product to purchase a Hotel Room once a year and have on hand a more expensive product they could compare against. It should absolutely be that easy!
It wouldn't matter where your from, # of distributions centers, etc.

[cjpremierfour]

The problem though is so what? Let's say 100 people listen to Product A and product B. Let's say 98 people like product B better.

Only 2 people liked product A better.

As a shopper you might play the odds and choose B because "most people" like it better. But most people might be clueless tone deaf dimwits. What if the 2 are Itzhak Stern and John Williams and the 98 are Keith Richards types only more stoned?

Absolutely right in a way! I attend audiophile shows very often and would like to think that most of audiophile community understands the basics of producing music over nonsense. If you let an average Joe start voting, it would destroy the whole concept.
Could Audiophile Forums gather in one place and get this process done correctly? I think that they could!

Even in my mini-article above - even with all the level matched blind testing hoopla - if it's not 100% it's not telling you anything.

I would still trust a large % number of audiophiles over a few magazines that tend to review the same equipment over and over. I also find that some of these reviewer have unlimited budgets that will allow them to put a pair of $50k Krell Mono blocks in their closet to take out for special occasions. I am not saying there is "Kick backs" involved, what I am saying is that I would like to take any chance of that happening out of the equation.

There is always going to be someone who likes a Big Mac over Gordon Ramsey burger, or SS over SET. A lack of good taste is abound :biggrin5:

"We just reviewed Gordon Ramsey's Cheeseburger last fall but now he has added a another slice of pickel, we are going to review it again". The extra pickel filled the deep spacious space in my stomach creating an excitement that I have never before experienced!
Does this sound familiar to anyone?

Again, that's the problem. The "Big Mac" can't get reviewed. The few magazines that do reviews will jump at certain manufacturers everytime there is a resistor change. (ie. series II / Model B) There are hundreds of companies left out of the chain that get no press except the internet.
As you stated earlier, who are these guys that gave the negatives and positives reviews. We need a reference and I think that can be done.

What most people believe doesn't make it so - ahem religion.
That's why we must assign a score. There has to be a winners and losers for this process to work.

[RGA]

The problem with assigning scores is that every individual weights things slightly differently - it can never be purely objective.

I am in the process of developing a numerical based system for reviewing products that takes the price out of the equation and then places the price back into the equation for a "value rating score."

Some magazines do this to some degree - Hi-Fi Choice rates things out of 100, enjoythemusic.com does too. Martin Colloms has a convoluted one.

The idea was to break things down into parts:

Ratings are in absolute terms rated against all speakers of all price ranges and ratings are not inflated because they are deemed budget speakers.
Deep Bass 10hz ? 30hz______________________0/10
Bass 30hz ? 80hz___________________________4/10
Midbass (80hz ? 200hz)______________________6.5/10X2
Midrange (200hz ? 3khz)_____________________ 7.5 /10 X3
High Frequency (3khz ? up)___________________7.5/10 X2
Attack (including macro-dynamics)_____________7/10
Decay____________________________________ 7.5/10
Resolution (including micro-dynamics)__________ 7/10 X2
Imaging and Soundstage_____________________ 7/10
Tonality___________________________________ 7.5/10 X2
Cohesiveness (integrity of the musical event)_____ 7.5/10 X4

Overall____________________________________135/200 = 67.5%

The overall score is the overall score. Then I factor in price. I had slots where a speaker at a given price point should have a score of a given number. This was a $600 speaker so this was a massive score and I awarded it a Best Buy tag. If however the speaker was $8000 then this score is dreadful because you are paying $8000 for a speaker that is no better than a $600 speaker.

Where there is a X2 or a X3 that is the added weight I give to each aspect of sound. I value cohesiveness as being 4 times more important than soundstaging or macrodynamic impact for example.

Another reviewer could use the exact same criteria but they may value soundstaging four times more than cohesiveness or tonality.

The idea though is to have an overall score and the reader can then look at the scores and say ok Richard gave the speaker a 10/10 for soundstaging - but because the reader will weight things differently the reader can then take the score and apply their own weighting to the score. So if I gave soundstage 10/10 and cohesiveness a 5/10 then out of 50 my score would 30/50. But if you value staging way more then your result would be 45/50.

In this way a review can be objective as possible about the quality of a given aspect of sound while also telling the reader what his personal bias or "values are" about what is important.

I think it would actually help readers spot which reviewers will be helpful to them on the one hand but also be helpful in getting some value from reviewers who you don't necessarily agree with but like their review style by simply overlaying your own weightings.

This was my first attempt at a number based review system - I am attempting to fix it up but have other articles and reviews on the go so it's in the percolating stage.

Here were my reviews back in 2004 on this website

http://forums.audioreview.com/speakers/rga-reviews-$1-000-00-loudspeakers-wharfedale-athena-technologies-etc-page-1-a-6694.html

http://forums.audioreview.com/speakers/rga-reviews-$2000-00-page-2-epos-audio-note-paradigm-b-w-etc-6696.html

http://forums.audioreview.com/speakers/rga-reviews-page-3-yes-still-more-6698.html

Man I've been raving on about the same speakers for over 7 years now. :yikes:

It's funny but the Audio Note AN K I raved about back then and has been selling since 1992 just won the best Standmount speaker award in Hi-Fi Review in Asia. I love being ahead of the curve sometimes.

[Mash]

Ah-so...............


Consumer Reports was actually correct when they developed their Loudspeaker Accuracy Score ..............