Open Call to Mtry, Skeptic, Naysayers, and ABXers

[magictooth]

Let me state from the outset that I am neither naysayer nor yeasayer. I want to come to my own conclusions about how different DVD players sound. I've tried to post a methodology that would allow me to be able to A/B different players at my home with as little fuss as possible. This methodolgy, however, has not met with the satisfaction of some members.

I realize that the title was perhaps misleading. Instead, I should have written, "Please judge my methodology and PROPOSE ANY REASONSABLE SOLUTIONS THAT CAN BE EASILY IMPLEMENTED AT HOME." I don't have access to an ABX machine, I have no access to a voltage attenuator at home although I do have a voltage reader. My amplifier is an Audiomat Solfege Reference. This unit has an analog knob for volume control and a switch for mute. That's it and that's all. There are no notches to be able to accurately level match the inputs when they get changed.

Before I detail once again the methodology that I am proposing, I have a question for MTry, Skeptic, Naysayers, and other ABXers. A simple yes/no answer in your reply is greatly appreciated. The question is as follows:

HAVE YOU EVER DONE AN A/B COMPARISON BETWEEN CD OR DVD PLAYERS YOURSELF?

I'm interested in the above question because I would like to have replies from people who have actually practiced what they preach. If you have done an A/B comparison personally, what methodology did you use? How did you completely eliminate bias? If you haven't done A/B tests personally, I would like to know how you came to the conclusion that all well designed CD/DVD players sound exactly the same, and that you shouldn't spend over $XXX.00 dollars on a machine.

The methodology that I am proposing is as follows:

1) 2 identical discs with a variety of material.
2) 2 separate players running in synchronous time (as much as I can allow) with displays covered.
- the units that I have right now are Integra Research RDV-1 (($1000/$3000), Classe DVD-1($1300/$3500), Toshiba SD-1600 ($20/$250), Toshiba SD-3950 ($NA/$80) The prices are (NEW/USED).
3) Identical cables running into the integrated amplifier (Audiomat Solfege Reference).
4) I will listen to the music until I can identify which DVD player is playing.
5) The wife will click the mute switch and do the following: 1) She will change the volume knob or she will leave it the same. 2) She will change the input switch or she will leave it the same. In both cases, I will not have any idea of whether she changed volume or inputs or whether they stayed the same. I also won't have any idea if she changed one variable and left the other as is.
6) After an interval of between 10-15 seconds, the wife will turn the volume back on. This should eliminate any bias that there is any discrepancy in the synchronization of the units.

I would appreciate any help in trying to find a SOLUTION if this methodology is deemed poor. Comments that YOU MUST level match to within .1dB are not useful by themselves. A useful comment would be "this is how you attain good level matching by doing these easy steps...."

[Thomas_A]

Yes I've made tests.

In one trial there was four persons listening. The equipment is described here:

http://hem.bredband.net/b113928/nyabilder.htm

We tested a Marantz CD6000, a cheap Toshiba SD-220E DVD player and an Audiolab player. All players were tested against a Musical Fidelity X24-K DAC. The players and the DAC were in this test not level-matched, but the sync problem was not there because of the DAC/CD combo. After some extensive listening, two of the listeners gave up to find any difference. Two of them, including me, were able to discern a slight difference but this was most likely due to the level difference (?0.4 dB or so).

In a second trial using another system and the two persons that heard a slight difference tested another DAC, Enlightened Audio Designs DSP-1000 MkII, + the X-24K + the internal DAC of an expensive Classé HT preamplifier. The CD drive was a TEAC VRDS, and pre/power amp was high quality DIY projects (I've never heard so low noise levels from an amp, e.g. the Classé had much more noise in comparison). Speakers were high-quality DIY with very flat frequency response curve. In this case were were able to calibrate the levels better using the Classé preamp (close to 0.1 dB) using a scope. We listened but could not hear any difference.

A a third trial I published AIFF files from a song coming from three different DACs/players on an internet HiFi forum. Although this is not an optimal test, errors from the players should be additive and if there are differences, it should be able to be heard. No one could say which file that originated from which DAC/player.

Some more trials, in which one succeded:

http://www.jrsaudio.se/dbtoncdplayers.htm

Some strange measurements:

http://hem.bredband.net/b113928/CD_players.htm

T

[Thomas_A]

Regarding level matching, if there is no way of level match there is no good way of making any AB blind test. A separate DAC should be used for the record/sync problem.

One way that is more easy to do at home without level matching (but which is far more difficult to hear differences between players) is that the tester reduce the volume to zero, do the switch on the input and then inrease the volume slowly again until the listener is satisfied (by spoken commands from the listener only, e.g. "stop", "increase volume", "decrease", "switch"). There can be an agreement before the trials that each trial always starts using the "CD" input and then a switch to "Tuner" and back to "CD" etc. There must be no sound or spoken word from the test control leader during the entire procedure, meaning that the listener must keep track of the input that he/she is listening to. Also if possible visual contact should be avoided by e.g. having the test leader behind the listener during the trials. This can be repeated until the listener has determined which player that is connected to inputs "CD" and "Tuner".

The listener must then exit the room and the test leader makes the switching of cables at the inputs. Even if there is no switching according the the predetermined random scheme, the cables should be removed and put there again so there is no obvious time difference between each trial. The test leader says "ready" and then listener enters the room and the next trial starts.

Repeat 15 times and compare the result against the random scheme.

Far from a perfect test, but anyway.

T

[skeptic]

The best way to perform an AB comparison IMO is to listen to the same musical passages on each in fairly rapid succession. The there should be a wide variety of musical selections showing different aspects of the kind of music you like to listen to. There should be a wide variety of dynamics, instruments, diffferent combinations, and the recordings should be of high quality. The levels must be exactly matched. The selected passages should be long enough for you to get a flavor of each sample but not so long as to forget what the previous sample sounded like. All other variables should be ruled meaning that the tests should be repeated on a wide variety of equipment.

For a cd player, this means two identical discs synchronized between the players, a variable output with a volume control on at least one of the players, the louder of the two, A-B repeat, and several sound systems to try them out with. Synchronizing cds is probably easier than with any other program source except FM radio. If you want to make the test scientifically fair, someone else should do the switching and you should have several listeners with a large number of samples.

I have never done any such test myself. However, I have determined to my own satisfaction that there are subtle audible differences between some players. This was done by a much less formal process where I did the switching myself. I was curious to see how a very highly rated JVC 1 bit unit I bought for my parents stacked up against a 4 times as expensive 20 bit Denon uint I had bought for myself several years earlier. At that time, I concluded that the differences were slight having to do with the relative high frequency balance and what I call steeliness of violin strings, slight on the Denon, non existant on the JVC. The JVC was also slightly brighter. That was about 13 years ago. Ironically, this year, the Denon unit finally failed and I decided not to repair it but to replace it with the JVC unit. Slight adjustments to the high end response made the JVC sound about identical as to what I remembered from the Denon. Not scientific but adequate for my purposes.

I am extremely suspicious when someone says one cd player blew another one away. From what I can tell, the differences are always subtle. If you will not take any steps to change the frequency response of a sound system other than to experiment with expensive cables or reposition your loudspeakers, then the sonic signiture of one cd player due to its deviation from flat frequency response can add or detract from deviations inherent in the rest of a sound system to the benefit or detriment of overall sound differently from another unit. If that's what makes some people think brand x is better than brand y and therefore justifies 5x or 10x the cost, then let them pay the high price. Most of todays so called high fidelity loudspeakers suffer from being too bright in many rooms and a cd player with a slight high end rolloff may sound better. BTW, the current unit in my second system is a 5 year old 5 disc RS 1 bit carousel unit and it sounds just fine. My chief conmplaints, it has no volume control, no AB repeat, and won't display remaining time per track.

[mtrycraft]

Let me state from the outset that I am neither naysayer nor yeasayer. I want to come to my own conclusions about how different DVD players sound. I've tried to post a methodology that would allow me to be able to A/B different players at my home with as little fuss as possible. This methodolgy, however, has not met with the satisfaction of some members.

I realize that the title was perhaps misleading. Instead, I should have written, "Please judge my methodology and PROPOSE ANY REASONSABLE SOLUTIONS THAT CAN BE EASILY IMPLEMENTED AT HOME." I don't have access to an ABX machine, I have no access to a voltage attenuator at home although I do have a voltage reader. My amplifier is an Audiomat Solfege Reference. This unit has an analog knob for volume control and a switch for mute. That's it and that's all. There are no notches to be able to accurately level match the inputs when they get changed.

Before I detail once again the methodology that I am proposing, I have a question for MTry, Skeptic, Naysayers, and other ABXers. A simple yes/no answer in your reply is greatly appreciated. The question is as follows:

HAVE YOU EVER DONE AN A/B COMPARISON BETWEEN CD OR DVD PLAYERS YOURSELF?

I'm interested in the above question because I would like to have replies from people who have actually practiced what they preach. If you have done an A/B comparison personally, what methodology did you use? How did you completely eliminate bias? If you haven't done A/B tests personally, I would like to know how you came to the conclusion that all well designed CD/DVD players sound exactly the same, and that you shouldn't spend over $XXX.00 dollars on a machine.

The methodology that I am proposing is as follows:

1) 2 identical discs with a variety of material.
2) 2 separate players running in synchronous time (as much as I can allow) with displays covered.
- the units that I have right now are Integra Research RDV-1 (($1000/$3000), Classe DVD-1($1300/$3500), Toshiba SD-1600 ($20/$250), Toshiba SD-3950 ($NA/$80) The prices are (NEW/USED).
3) Identical cables running into the integrated amplifier (Audiomat Solfege Reference).
4) I will listen to the music until I can identify which DVD player is playing.
5) The wife will click the mute switch and do the following: 1) She will change the volume knob or she will leave it the same. 2) She will change the input switch or she will leave it the same. In both cases, I will not have any idea of whether she changed volume or inputs or whether they stayed the same. I also won't have any idea if she changed one variable and left the other as is.
6) After an interval of between 10-15 seconds, the wife will turn the volume back on. This should eliminate any bias that there is any discrepancy in the synchronization of the units.

I would appreciate any help in trying to find a SOLUTION if this methodology is deemed poor. Comments that YOU MUST level match to within .1dB are not useful by themselves. A useful comment would be "this is how you attain good level matching by doing these easy steps...."

Your muting duration should eliminate sync problems.
Changing volumes randomly to random discs may or may not be an issue. Hard to say without testing this methodology itself. Try it. It seems that you have a willing assistant?

Your muting duration will hinder your acoustic memory for small details between the two if there are any.

Not playing and comparing the same musical passages will hinder you as well.

Do a large number of trials, at least 20. She shoule either flip a coin or find a random table or the decimal digits of pi, they are random, as either even or odd and swith to A or B.

Make sure you keep a score card and your wife does too. Compare after the test only.
No, I have not compared CD players DBT; no reason for me to do.

[996turbo]

No, I have not compared CD players DBT; no reason for me to do.
__________________

All the more reason to get together.

Only a fool follows

[mtrycraft]

No, I have not compared CD players DBT; no reason for me to do.
__________________

All the more reason to get together.

Only a fool follows

Why, for what reason? You have credible evidence that there is a need? No, your story is not credible evidence of anything, just a unreliable story.

[996turbo]

All I am saying Mtrycraft is that personal experience is important. One can read things in books and take them for fact. Have you ever experienced things in real life that were different than what you read or expected? Think about all the advancements in life that would never have occured if we blindly followed. I have read how wonderful or on the contrary bad something could be but was suprised when I finally experienced it.

All I am asking is for you to open your eyes and at least try and experience what the other side is experiencing. You may be suprised, or is that what you are concerned about?

I tell my 6 year old daughter all the time to try something that you do not like you may be suprised and like it. You know what sometimes she does and sometimes she does not but at least she has the experience to prove that she did or did not.

[Thomas_A]

996turbo,

likewise, experience from many open and blind tests would be experience too, don't you agree? And "obvious" differences that disappear as soon as the test shifts to blind conditions, that is experience isn't it?

And there are many people shifting from one side to the other when they have been conducting such tests, and realizing that they previously had been imagining the differences they heard during open conditions. That is being open minded isn't it?

To have "open eyes" means performing critical tests (sighted and blinded) and draw the conclusion from their results. The work by Floyd E. Toole is a good starting point.

T

[mtrycraft]

All I am saying Mtrycraft is that personal experience is important.


Not if it is based on flawed and unreliable testing protocols. Might as well not have them; has no meaning. No extra credit.

One can read things in books and take them for fact. Have you ever experienced things in real life that were different than what you read or expected?

All depends on the book. Magic books? Fiction? Or real science books?

Think about all the advancements in life that would never have occured if we blindly followed.

Think of all the time wasted trying to reinvent the wheel.

I have read how wonderful or on the contrary bad something could be but was suprised when I finally experienced it.

Totally situational. Maybe it was placebo at work.

All I am asking is for you to open your eyes and at least try and experience what the other side is experiencing.

Oh, my eyes are very open. What others are experiencing have not been deomnstrated to be a fact as far as audible differences we are yalking about. So, those exepriences are only a singular reality, fantasy in another word.

You may be suprised, or is that what you are concerned about?

Not at all.

I tell my 6 year old daughter all the time to try something that you do not like you may be suprised and like it.

Ah, this isn't a dislike of tastes, or anything. This is something that can be tested properly which you have not, so your claims are baseless.

[skeptic]

CD players are easy to sync using the cue function buttons <- ->. However, with two cd players set for AB repeat, even that is not necessary. With both players set to A-B repeat the same passage, all you have to do is reset one to the beginning of the AB segment <<-- while the other is playing. It is not a fair test if the two players aren't playing the same passage IMO.

I don't know what this would prove. That one player sounds different from the other is about all you would likely discern. That one player sounded better but not necessarily more accurate playing one disc on one sound system. I still don't understand what conclusions you would come to.

Early players had gross distortions. All 16 and 18 bit units as late as 1989 reproduced violins with a metalic steely sound. The 20 bit Denon was far better in this respect. By 1991, the single bit chips changed all that. The JVC unit convinced me and by 1992, a portable Sony car Discman 808DK ($300) sounded just as good. Minor frequency response differences if they are at audible at all do not concern me.

The mechanical "quality" build of the expensive Denon player was not translated into greater reliability. It had nothing but problems almost from the start. Three trips to the nearest authroized service about 20 miles from home to repair the misaligned clamper arm were fruitless and in the end, I had to shim it myself to get it to work properly on all discs. The metal parts in the transport while heavy and well machined proved no better in use than the much cheaper plastic units. No other transport was as troublesome. The first time I disassembled it, I found out that most of the weight didn't come from the transport or a large oversized power transformer but sheet steel plates on the top and bottom. Like many other "early" high quality electronics products such as the $1500 Panasonic VCR I once owned, it served its purpose in its day but it won't be missed. It had a lot of useless features as well that I never used.

[magictooth]

Well, thanks to you three for your insights and anecdotes. I will try to do some A/Bs with your tips in mind. I'll leave the room for input and volume changes and I'll have the wife leave so that there's no visible cues at all. Of course we won't be comparing while the tests are going on - only afterwards.

[Thomas_A]

As a first try, it would not be nessecary to move from the room during the trial. If you sit in the sofa/chair and state the commands, e.g. switch:

And your wife lowers the volume to zero make the switch from CD to Tuner input and raise volume again until you say "stop". She can adjust the volume if you say higher or lower, but it is important that you make the choices, not she.

At the next command "switch" from you, she repeat the procedure but switch bac to CD. And this can continue for some time, until you write down what is on "CD" and on "Tuner". You can make notes during the time so you don't mix up the inputs and are free to listen for how long time you want.

When you leave the room, that is the signal for your wife to switch cables according to a random scheme. When she is ready, trial number 2 starts.



T