Finally Finished My A/B Trials...Here're the Results

[magictooth]

Well, I finally got around to doing the blind A/B trials that I had been planning to do for a while. I won't bore you with the details of the methodology which can be found here: Please Judge My A/B Methodology, Mtry Others.

I did add another wrinkle into the trials. My first A/B was a true David vs. Goliath match up with the Toshiba SD-1600 going up against the Classe DVD/CD-1. The first is valued probably around $0-20 while the second on the second hand market is around $1200-1500.

My results with this match up were likely a bit surprising to the naysayers. I could tell the difference each and every time. There was no contest between the two. In fact, you could say it was "a night and day difference." Now you'd think that I'd just stop here and proclaim victory for the yeasayers, but I'm not sure how well a 5 year old 5 pound DVD player is going to work after such a long time. I'd be willing to wager that the normal useful lifespan of this player has long been exceeded.

Hence, the second A/B trial that I did was with a new Toshiba SD-3950 that I picked up at Walmart for $89Cdn. This unit purports to have 192/24 DACs and to have the newest Crystal decoding chip. It would be interesting to see how this new unit stacked up against the slightly older Classe model.

I did the A/B and I was a bit surprised at the results. In 6 separate trials with different types of music, I found that the Toshiba and the Classe were virtually indistinguishable from each other. I say virtually because there was one selection where a difference could be detected. This selection is the Mozart Flute and Harp concerto. The Harp when played on the Toshiba had a digital type of edge to it; when played on the Classe, it sounded more like a harp. I could tell the difference between the two units in this one case.

In all the other cases, such as with Gershwin, Bruch violin concerto, Dvorak cello concerto, Norah Jones, and Stone Temple Pilots, I could tell no difference at all between the two units. I had what you could consider a very poor result if you are a yeasayer. I got 14/25 tries correct and I'm sure that this means nothing at all given the small statistcal sample. For the harp selection, I scored 5/5, but I also realize that the small sample size may not mean a whole lot.

In any case, I'm not here to argue statistical significance, but to let people know that the results that I got from my A/B trials showed that buying a very expensive CD or DVD player may not result in as good an improvement in sound as you'd like to see - especially considering how many times more you've spent on the unit compared to a mass market model.

My final conclusion is that there is a difference between different CD/DVD players, but, for me, the difference is not sufficient to justify the multiplication of cost.

[Thomas_A]

Well, I finally got around to doing the blind A/B trials that I had been planning to do for a while. I won't bore you with the details of the methodology which can be found here: Please Judge My A/B Methodology, Mtry Others.

I did add another wrinkle into the trials. My first A/B was a true David vs. Goliath match up with the Toshiba SD-1600 going up against the Classe DVD/CD-1. The first is valued probably around $0-20 while the second on the second hand market is around $1200-1500.

My results with this match up were likely a bit surprising to the naysayers. I could tell the difference each and every time. There was no contest between the two. In fact, you could say it was "a night and day difference." Now you'd think that I'd just stop here and proclaim victory for the yeasayers, but I'm not sure how well a 5 year old 5 pound DVD player is going to work after such a long time. I'd be willing to wager that the normal useful lifespan of this player has long been exceeded.

Hence, the second A/B trial that I did was with a new Toshiba SD-3950 that I picked up at Walmart for $89Cdn. This unit purports to have 192/24 DACs and to have the newest Crystal decoding chip. It would be interesting to see how this new unit stacked up against the slightly older Classe model.

I did the A/B and I was a bit surprised at the results. In 6 separate trials with different types of music, I found that the Toshiba and the Classe were virtually indistinguishable from each other. I say virtually because there was one selection where a difference could be detected. This selection is the Mozart Flute and Harp concerto. The Harp when played on the Toshiba had a digital type of edge to it; when played on the Classe, it sounded more like a harp. I could tell the difference between the two units in this one case.

In all the other cases, such as with Gershwin, Bruch violin concerto, Dvorak cello concerto, Norah Jones, and Stone Temple Pilots, I could tell no difference at all between the two units. I had what you could consider a very poor result if you are a yeasayer. I got 14/25 tries correct and I'm sure that this means nothing at all given the small statistcal sample. For the harp selection, I scored 5/5, but I also realize that the small sample size may not mean a whole lot.

In any case, I'm not here to argue statistical significance, but to let people know that the results that I got from my A/B trials showed that buying a very expensive CD or DVD player may not result in as good an improvement in sound as you'd like to see - especially considering how many times more you've spent on the unit compared to a mass market model.

My final conclusion is that there is a difference between different CD/DVD players, but, for me, the difference is not sufficient to justify the multiplication of cost.

Thank you for making the effort to do these tests. I agree with you that it is difficult to hear any differences between players. Sometimes, between certain pairs of CD players it is possible, but during normal music listening at home, very difficult.

Thomas

[okiemax]

Well, I finally got around to doing the blind A/B trials that I had been planning to do for a while. I won't bore you with the details of the methodology which can be found here: Please Judge My A/B Methodology, Mtry Others.

I did add another wrinkle into the trials. My first A/B was a true David vs. Goliath match up with the Toshiba SD-1600 going up against the Classe DVD/CD-1. The first is valued probably around $0-20 while the second on the second hand market is around $1200-1500.

My results with this match up were likely a bit surprising to the naysayers. I could tell the difference each and every time. There was no contest between the two. In fact, you could say it was "a night and day difference." Now you'd think that I'd just stop here and proclaim victory for the yeasayers, but I'm not sure how well a 5 year old 5 pound DVD player is going to work after such a long time. I'd be willing to wager that the normal useful lifespan of this player has long been exceeded.

Hence, the second A/B trial that I did was with a new Toshiba SD-3950 that I picked up at Walmart for $89Cdn. This unit purports to have 192/24 DACs and to have the newest Crystal decoding chip. It would be interesting to see how this new unit stacked up against the slightly older Classe model.

I did the A/B and I was a bit surprised at the results. In 6 separate trials with different types of music, I found that the Toshiba and the Classe were virtually indistinguishable from each other. I say virtually because there was one selection where a difference could be detected. This selection is the Mozart Flute and Harp concerto. The Harp when played on the Toshiba had a digital type of edge to it; when played on the Classe, it sounded more like a harp. I could tell the difference between the two units in this one case.

In all the other cases, such as with Gershwin, Bruch violin concerto, Dvorak cello concerto, Norah Jones, and Stone Temple Pilots, I could tell no difference at all between the two units. I had what you could consider a very poor result if you are a yeasayer. I got 14/25 tries correct and I'm sure that this means nothing at all given the small statistcal sample. For the harp selection, I scored 5/5, but I also realize that the small sample size may not mean a whole lot.

In any case, I'm not here to argue statistical significance, but to let people know that the results that I got from my A/B trials showed that buying a very expensive CD or DVD player may not result in as good an improvement in sound as you'd like to see - especially considering how many times more you've spent on the unit compared to a mass market model.

My final conclusion is that there is a difference between different CD/DVD players, but, for me, the difference is not sufficient to justify the multiplication of cost.

Thanks for doing the test and sharing the results. Most forum members, including myself, talk and argue about testing, but rarely do any. It's good to see someone making an effort. I don't know whether the test results suggest the Toshiba SD-3950 is an over achiever or the Classe DVD/CD-1 isn't much better than low-priced players. Loving a bargin, I would hope for the fomer, but my guess would be the latter. Is the Toshiba something special?

[mtrycraft]

Yes, thanks for the report. Interesting

[E-Stat]

I don't know whether the test results suggest the Toshiba SD-3950 is an over achiever...

The Tosh is an overachiever. I use one in the HT system. By comparion, my GamuT CD-1 does not sound forty time better. Naturally, part of the price differential is due to the disposable-plastic build quality of the Tosh vs. the GamuT. The thick anodized aluminum faceplate of the CD-1 alone costs more than the entire Toshiba unit.

As an aside, Toshiba also has a universal player, the 4960, for just a few bucks more. That may prove to be among the best player bargains.

rw

[Geoffcin]

Well, I finally got around to doing the blind A/B trials that I had been planning to do for a while. I won't bore you with the details of the methodology which can be found here: Please Judge My A/B Methodology, Mtry Others.

I did add another wrinkle into the trials. My first A/B was a true David vs. Goliath match up with the Toshiba SD-1600 going up against the Classe DVD/CD-1. The first is valued probably around $0-20 while the second on the second hand market is around $1200-1500.

My results with this match up were likely a bit surprising to the naysayers. I could tell the difference each and every time. There was no contest between the two. In fact, you could say it was "a night and day difference." Now you'd think that I'd just stop here and proclaim victory for the yeasayers, but I'm not sure how well a 5 year old 5 pound DVD player is going to work after such a long time. I'd be willing to wager that the normal useful lifespan of this player has long been exceeded.

Hence, the second A/B trial that I did was with a new Toshiba SD-3950 that I picked up at Walmart for $89Cdn. This unit purports to have 192/24 DACs and to have the newest Crystal decoding chip. It would be interesting to see how this new unit stacked up against the slightly older Classe model.

I did the A/B and I was a bit surprised at the results. In 6 separate trials with different types of music, I found that the Toshiba and the Classe were virtually indistinguishable from each other. I say virtually because there was one selection where a difference could be detected. This selection is the Mozart Flute and Harp concerto. The Harp when played on the Toshiba had a digital type of edge to it; when played on the Classe, it sounded more like a harp. I could tell the difference between the two units in this one case.

In all the other cases, such as with Gershwin, Bruch violin concerto, Dvorak cello concerto, Norah Jones, and Stone Temple Pilots, I could tell no difference at all between the two units. I had what you could consider a very poor result if you are a yeasayer. I got 14/25 tries correct and I'm sure that this means nothing at all given the small statistcal sample. For the harp selection, I scored 5/5, but I also realize that the small sample size may not mean a whole lot.

In any case, I'm not here to argue statistical significance, but to let people know that the results that I got from my A/B trials showed that buying a very expensive CD or DVD player may not result in as good an improvement in sound as you'd like to see - especially considering how many times more you've spent on the unit compared to a mass market model.

My final conclusion is that there is a difference between different CD/DVD players, but, for me, the difference is not sufficient to justify the multiplication of cost.

You busted your butt and found out that good players are easy to spot with good recordings, nearly all of the time.You also found out the nasty other thing; That 90%-95% of all CD recorded music will not be served up any better between most CD players.

Is a Z06 Corvette worth 5x a Hundai? Not if your only going to drive on pot-holed roads to the supermarket. Or even just around town. On the few roads that will let you do what it's capable of, then it's magic. Is it worth it? To some people, the answer is obviously yes.

[Thomas_A]

You busted your butt and found out that good players are easy to spot with good recordings, nearly all of the time.You also found out the nasty other thing; That 90%-95% of all CD recorded music will not be served up any better between most CD players.

Easy to spot with good recordings? I don't see where that is written in this test.

[NickWH]

Unless I'm blind, I didn't see a list of associated equipment used for these "tests." I only see references to the actual players. Just curious...

[Beckman]

Hence, the second A/B trial that I did was with a new Toshiba SD-3950 that I picked up at Walmart for $89Cdn. This unit purports to have 192/24 DACs and to have the newest Crystal decoding chip. It would be interesting to see how this new unit stacked up against the slightly older Classe model....My final conclusion is that there is a difference between different CD/DVD players, but, for me, the difference is not sufficient to justify the multiplication of cost.

Thank you for taking the time to run a test. I am not suprised by the results, kind of wished I would have known this before purchasing my $400 Cambridge D500SE. Oh well, at least I helped out the economy.
Hmmm,
$89 vs. $1200.

[mtrycraft]

Unless I'm blind, I didn't see a list of associated equipment used for these "tests." I only see references to the actual players. Just curious...

That would have just satisfied some and dissapointed others. Besides, it is not that relevant.

[mtrycraft]

Oh well, at least I helped out the economy.
Hmmm,
$89 vs. $1200.

Yes, that is a worthwhile endevor when it is other people's money

[magictooth]

Unless I'm blind, I didn't see a list of associated equipment used for these "tests." I only see references to the actual players. Just curious...

I didn't think that it was that important, but the equipment used was as follows:

Audiomat Solfege Reference
Tannoy Studio 12 DMT II monitors
Some type of customs ICs ($15/pair)
Silver something or other speaker wire ($2.50/ft)

[Resident Loser]

...and it really isn't...there is that contingent, however, who will pooh-pooh any results that seem to erode their belief system...your speakers, your electronics...he!! the paint color in your listening room...all of these things can be grasped at to cause discredit.

Is their any guess as to why mtry barely lays claims to listening to a boombox...

jimHJJ(...note, I didn't say owns one...)

[E-Stat]

Is their any guess as to why mtry barely lays claims to listening to a boombox...

Guess not required. It is quite evident. Real science and curiosity does not hide from the truth. While I may not share the same viewpoints as Skeptic, I admire his openness to answer such fundamental questions. Mtry is a good parrot though.

(...note, I didn't say owns one...)

He claims he does. Here are but two references.

One

Two

rw

[NickWH]

didn't think that it was that important

Actually, it is very important. Without resolving and detailed equipment, how could one determine one component from another? I'm not saying this is the case here, just in general.

I have found in the past few years in my pursuit of happines that I have found much more joy out of helping others with no expectation of anything in return than I have out of spending large sums of money on stereo equipment, but that is just me.

Great, you deserve a medal.

[Beckman]

Great, you deserve a medal.

I appologize for my post. I have deleted it as not to offend anyone else.

[mtrycraft]

Guess not required. It is quite evident. Real science and curiosity does not hide from the truth. While I may not share the same viewpoints as Skeptic, I admire his openness to answer such fundamental questions. Mtry is a good parrot though.



He claims he does. Here are but two references.

One

Two

rw

Hey, I might even have two boomboxes, so what? Does that matter what YOU can demonstrate being audible? UNder bias controlled conditions? Didn't think so.

[mtrycraft]

Actually, it is very important. Without resolving and detailed equipment, how could one determine one component from another? I'm not saying this is the case here, just in general. .

Ah, yet another unsubstantiated claim making for component needs. I am sure evidence is plenty which is more resolving and which is not. Or, that is just a subjective guess on ones part?

[NickWH]

Or, that is just a subjective guess on ones part?

Look, I don't claim to know how to measure these things. I just know what I hear. Are YOU saying that it wouldn't matter which amplifier or speakers were used during these A/B tests? All I'm saying is that it DOES matter. There is nothing subjective about that.

[Resident Loser]

...it's just an ego stroke...it makes not one bit of difference because they are not the items under test...even less so to the participant...he or she is aware of the systems sound under normal circumstances. What is being examined is the new variable...

It's the same ol' same ol', "you don't know how to listen"..."you don't know what to listen for"..."your system hasn't enough resolution"...well he!! my system doesn't have pixels! So what exactly is sonic "resolution" anywho? Another one of those bits of so-called "golden-eared" jargon...like "burn-in" and all the rest...and puh-leease, don't give me the "inner detail" cr@pola! It's nondescript, meaningless verbiage; a convenient escape route for the mystics among us...

It's like the other thread about "red dawns" vs. "blue b@lls" or whatever..."...this wire can be too revealing..." Pure unadulterated claptrap!

jimHJJ(...get out the hip-boots boys, gettin' awful deep 'round here!...)

[Beckman]

...it makes not one bit of difference because they are not the items under test...even less so to the participant...he or she is aware of the systems sound under normal circumstances. What is being examined is the new variable...

I agree, except for one thing. Frequency response. The variable (in this case the cd player) could affect low end sound. If you are listening with speakers that only go down to 70 Hz you might not notice a change.

[okiemax]

...it's just an ego stroke...it makes not one bit of difference because they are not the items under test...even less so to the participant...he or she is aware of the systems sound under normal circumstances. What is being examined is the new variable...

It's the same ol' same ol', "you don't know how to listen"..."you don't know what to listen for"..."your system hasn't enough resolution"...well he!! my system doesn't have pixels! So what exactly is sonic "resolution" anywho? Another one of those bits of so-called "golden-eared" jargon...like "burn-in" and all the rest...and puh-leease, don't give me the "inner detail" cr@pola! It's nondescript, meaningless verbiage; a convenient escape route for the mystics among us...

It's like the other thread about "red dawns" vs. "blue b@lls" or whatever..."...this wire can be too revealing..." Pure unadulterated claptrap!

jimHJJ(...get out the hip-boots boys, gettin' awful deep 'round here!...)

The associated equipment is relevant to generalizations about the DVD/CD players. But I think the tester's comments were limited to how he thought the three players performed in his system, which he later described. He didn't say the results would have been the same regardless of the speakers, amps, etc. Nor did he say everyone would agree with his results using his system.

[Resident Loser]

...anyone who has a system that has usable output to only 70Hz has bigger problems than finding time to do A/B testing on CD/DVD players...they should be finding new speakers or at very least a decent sub.

Lo-freqs or FR in general in the purely electronic components isn't usually much of an issue...20Hz-20kHz is no real biggie...even the lower end "BB" and "CC" stuff can bench test those numbers...it's all those unmeasurable catchphrase qualities that are...a noted wire maven(conspicuous by his absence) is constantly downplaying FR as a minor player in the whole game, hence the elusive "resolution" bit...

Then of course, very few of us have the experience of hearing the "live" performance and rely only on the recording of it as a basis for judgement...so what are all of these ephmeral claims of rez and such...as far as digital is concerned it's all numbers...ones and zeros...on/off...yes or no...it's only when the analog stages are tinkered with that we get what we get...who is to say what is accurate and what is a distortion, it's all relative...you can "boost" the bass by attenuating the treble and you can sacrifice "accuracy" by romanticizing the sound.

It's funny, what I personally would consider "accurate" and hi-rez, reveals all the warts and whiskers. There are those who might consider it harsh. Many of these same folks refer to "resolution" and systems that are "revealing". IMO, they are usually highly colored or at least canted towards an imbalanced and distorted presentation.

jimHJJ(...Bit of a conundrum that!...)

[Beckman]

...anyone who has a system that has usable output to only 70Hz has bigger problems than finding time to do A/B testing on CD/DVD players...they should be finding new speakers or at very least a decent sub.

Lo-freqs or FR in general in the purely electronic components isn't usually much of an issue...20Hz-20kHz is no real biggie...even the lower end "BB" and "CC" stuff can bench test those numbers...)

I agree, but you don't know unless you ask. Someone might be comparing cd players using one of these for a speaker:

http://www.fcsurplus.ca/shopping/sho...OVMTC=standard

[magictooth]

I agree, except for one thing. Frequency response. The variable (in this case the cd player) could affect low end sound. If you are listening with speakers that only go down to 70 Hz you might not notice a change.

Not sure if you're referring to my speakers or if you're just making a general statement, but the speakers that I have are rated +/-3dB from 40-20K Hz. I've also got a sub (which I forgot to list) that is set up via variable crossover at about 40 Hz. The reason why I forgot is that it rarely ever turns on. I think that most music is played above 40 Hz although I realize that there are a ton of exceptions. I think that it might have gotten some use when STP was playing, but otherwise it just sits there quiet as as a mouse.