Hey Mark. Are A/B tests worthless?

[pctower]

Here's what one obvious "ignoramous" said on the subject:

"I've come to the conclusion that short term AB tests are pretty much worthless. I have one anecdotal story that illustrates this particularly well. A friend of mine and I compared his $4000 Audio Research preamp with my $400 Audible Illusions preamp for an afternoon, swapping the two in and out of his system. We both agreed that the differences were subtle, perhaps too subtle to justify the staggering difference in price. One summer, he left for England and lent me his SP-10. I listened to it for a whole summer and really enjoyed my stereo. When he came back and I replaced the SP-10 with my Audible Illusions preamp, my system suddenly sounded lifeless and I didn't enjoy it as much. This puzzled me because on the AB test we both agreed the differences were subtle, yet the long term pleasure somehow wasn't even close.

I shortly thereafter went out and bought an SP-11 (the newest model at the time, listing for about $5000). When my friend sent his SP-10 back to the factory to be blessed or retubed or something, he borrowed my Audible Illusions, thinking that it would tide him over. I hadn't told him how disappointing I found it after the SP-10. He called me up a few weeks later to say that he was confused and that his system wasn't giving him any pleasure, that he was hardly listening to it. How could it sound so lifeless when an AB test proved nearly inconclusive?

An interesting question is why we ever thought an A/B test was so great to begin with. After all, would you evaluate two performances of Beethoven's Waldstein sonata in 1-minute snippets? If there were a magic machine that could zap you into an Acura NSX or a Chevy Caprice for two minute intervals, would you test the cars that way or would you live with each one for a week? And what about mates? Have you ever had a perfectly pleasant date with a person only to discover that marriage to the very same person wasn't such a great idea? "

See: http://philip.greenspun.com/materialism/stereo

But I'm sure Professor Greenspun's academnic credentials pale in comparison to yours and the other esteemed scientists on this board.

Pity, though, because we'll never know. Professor Greenspun believes in full disclosure. You and the other "scientists" here don't.

AH YES! My venom is really flowing good today.

[markw]

Here's what one obvious "ignoramous" said on the subject:

"I've come to the conclusion that short term AB tests are pretty much worthless. I have one anecdotal story that illustrates this particularly well. A friend of mine and I compared his $4000 Audio Research preamp with my $400 Audible Illusions preamp for an afternoon, swapping the two in and out of his system. We both agreed that the differences were subtle, perhaps too subtle to justify the staggering difference in price. One summer, he left for England and lent me his SP-10. I listened to it for a whole summer and really enjoyed my stereo. When he came back and I replaced the SP-10 with my Audible Illusions preamp, my system suddenly sounded lifeless and I didn't enjoy it as much. This puzzled me because on the AB test we both agreed the differences were subtle, yet the long term pleasure somehow wasn't even close.

I shortly thereafter went out and bought an SP-11 (the newest model at the time, listing for about $5000). When my friend sent his SP-10 back to the factory to be blessed or retubed or something, he borrowed my Audible Illusions, thinking that it would tide him over. I hadn't told him how disappointing I found it after the SP-10. He called me up a few weeks later to say that he was confused and that his system wasn't giving him any pleasure, that he was hardly listening to it. How could it sound so lifeless when an AB test proved nearly inconclusive?

An interesting question is why we ever thought an A/B test was so great to begin with. After all, would you evaluate two performances of Beethoven's Waldstein sonata in 1-minute snippets? If there were a magic machine that could zap you into an Acura NSX or a Chevy Caprice for two minute intervals, would you test the cars that way or would you live with each one for a week? And what about mates? Have you ever had a perfectly pleasant date with a person only to discover that marriage to the very same person wasn't such a great idea? "

See: http://philip.greenspun.com/materialism/stereo

But I'm sure Professor Greenspun's academnic credentials pale in comparison to yours and the other esteemed scientists on this board.

Pity, though, because we'll never know. Professor Greenspun believes in full disclosure. You and the other "scientists" here don't.

AH YES! My venom is really flowing good today.


Now, please point out to me where I've ever said that short term tests are the end all and be all.

Be honest now. I know that's a challange for you, but, please, give it a try and answer the question directly. ...without trying to change the subject as you're wont to do.

[WmAx]

Here's what one obvious "ignoramous" said on the subject:


I commend you on the accurate description. :-P

-Chris

[pctower]

I commend you on the accurate description. :-P

-Chris

Oh really! Would you care to post your resume so that we may all be enlightened as to how much more knowledgeable and accomplished you are than Professor Greenspun.

[pctower]

Now, please point out to me where I've ever said that short term tests are the end all and be all.

Be honest now. I know that's a challange for you, but, please, give it a try and answer the question directly. ...without trying to change the subject as you're wont to do.

OK Mark. Fair's fair. I can point to no place where you have ever said that.

Now, my challenge to you. In the bi-wire thread I brought up the subject of DBTs early on and you continued with that discussion. I don't consider that changing the subject, but reasonable men can differ.

However, you statement is that I am "wont" to change the subject, implying that is my normal pattern. Please substantiate your accusation.

[WmAx]

Oh really! Would you care to post your resume so that we may all be enlightened as to how much more knowledgeable and accomplished you are than Professor Greenspun.You believe because he is a professor, that he must be correct? I really don't care what his resume looks like, his statements on his link are presumptious. Lots of claims. No substantiation.

In reference too his preamp difference claims, their are two possibilities: (1) a measurable difference is present that is of audible significance [and/or] (2) pyschological bias.

-Chris

[markw]

OK Mark. Fair's fair. I can point to no place where you have ever said that.

Now, my challenge to you. In the bi-wire thread I brought up the subject of DBTs early on and you continued with that discussion. I don't consider that changing the subject, but reasonable men can differ.

However, you statement is that I am "wont" to change the subject, implying that is my normal pattern. Please substantiate your accusation.

Actually, I didn't bring up the subject of DBT's. You did. If you reread that thread again, you'll notice that my next didn't bring up DBTs either. n fact, as was pointed out several times later in that thread, you were the one constantly bringing up DBT's.

The thread was initiated on bi-wiring and all either of our posts did was to tell the poor guy that there were several schools of thought on that matter. Immediately after my post, you started in with me on the DBT issue.

In fact, in every one of your subsequent posts, even with the lawyer jokes as the focus, you always went back to the DBT issue? Why?

Now, had you chosen to get back to the issues at hand, (bi-wiring, sources of info, etc...) perhaps that ugly brouhaha might not have ensued.

But no, you always came back to DBT's.

And now you want to start a thread on them?

In all honesty and not to be insulting, I really think you have issues with DBT's. Can't a discussion evolve without DBT's? It seems that if nobody else brings it up, you do.

[pctower]

Actually, I didn't bring up the subject of DBT's. You did. If you reread that thread again, you'll notice that my next didn't bring up DBTs either. n fact, as was pointed out several times later in that thread, you were the one constantly bringing up DBT's.

The thread was initiated on bi-wiring and all either of our posts did was to tell the poor guy that there were several schools of thought on that matter. Immediately after my post, you started in with me on the DBT issue.

In fact, in every one of your subsequent posts, even with the lawyer jokes as the focus, you always went back to the DBT issue? Why?

Now, had you chosen to get back to the issues at hand, (bi-wiring, sources of info, etc...) perhaps that ugly brouhaha might not have ensued.

But no, you always came back to DBT's.

And now you want to start a thread on them?

In all honesty and not to be insulting, I really think you have issues with DBT's. Can't a discussion evolve without DBT's? It seems that if nobody else brings it up, you do.

I make no bones about having an issue with the claims that are made here based on supposedly valid or reliable tests that have been run in the past.

However, I note that you have totally ducked my request that you document your accusation that I am "wont" to change the subject. Simply expanding the scope of your accusation as you just did in your most recent post doesn't constitute documentation. It simply constitutes expanding the scope of your unsubstantiated allegation.

[pctower]

You believe because he is a professor, that he must be correct? I really don't care what his resume looks like, his statements on his link are presumptious. Lots of claims. No substantiation.

In reference too his preamp difference claims, their are two possibilities: (1) a measurable difference is present that is of audible significance [and/or] (2) pyschological bias.

-Chris

I didn't say I believed him - you made that up in your own mind. I simply asked Mark a question.

I note you have a big, loud, accusatory mouth, but refuse to tell us anything about yourself.

Well, let’s see what we can tell about you from your own words:

You believe because he is a professor, that he must be correct? I really don't care what his resume looks like, his statements on his link are presumptious (sic). Lots of claims. No substantiation.

In reference too (sic) his preamp difference claims, their (sic) are two possibilities: (1) a measurable difference is present that is of audible significance [and/or] (2) pyschological (sic) bias.

From that alone, we might suspect that you are either very careless or a terrible speller. I’ll bet Philip Greenspun would not let something like that go out with his name attached.

[WmAx]

I didn't say I believed him - you made that up in your own mind. I simply asked Mark a question.

I note you have a big, loud, accusatory mouth, but refuse to tell us anything about yourself.

Well, let’s see what we can tell about you from your own words:

You believe because he is a professor, that he must be correct? I really don't care what his resume looks like, his statements on his link are presumptious (sic). Lots of claims. No substantiation.

In reference too (sic) his preamp difference claims, their (sic) are two possibilities: (1) a measurable difference is present that is of audible significance [and/or] (2) pyschological (sic) bias.

From that alone, we might suspect that you are either very careless or a terrible speller. I’ll bet Philip Greenspun would not let something like that go out with his name attached.Slick. Real slick. True, you did not claim to believe him. But, your statements in the original post, do seem to infer that you did. As far as my spelling, not really relevant to the issue(s) at hand. Since we get to play the 'if I was him game', let me have my turn. :-) If I was Philip Greenspun, I would rather be guilty of careless spelling as opposed to the multitude of statements made as facts on his site, that are not established as such.

-Chris

[markw]

I make no bones about having an issue with the claims that are made here based on supposedly valid or reliable tests that have been run in the past.

However, I note that you have totally ducked my request that you document your accusation that I am "wont" to change the subject. Simply expanding the scope of your accusation as you just did in your most recent post doesn't constitute documentation. It simply constitutes expanding the scope of your unsubstantiated allegation.

Well, I see you’re back to playing games again by you denying your own actions. Phil, if you would even bother to re-read that last long thread on bi-wiring, you will plainly see where you were the one to constantly keep dragging up DBT’s, again and again.

Here's the kicker, guy. The thread was about bi-wiring, not DBT's. Who constantly tried to force the focus back to DBT's? Not I.

It's like you are one trick pony. All you want to do is argue DBT's, no matter what the subject at hand. And, in some fashion or another, you will try to bend the subject to your liking. ...and you wonder why I accuse you of acting like a liar, er... lawyer.

Now, if your own words are not "substantiation" enough for you, then I guess it’s a lost cause trying to carry on a meaningful discssion with you. So much for a meaningful discussion, eh?

Oh well, I guess it’s back to lawyer jokes.

An attorney was cross-examining a coroner.

The attorney asks, "before you signed the death certificate, did you take the man's pulse?" The coroner says, "No."

The lawyer then asks, "Did you listen for a heartbeat?" "No" says the coroner.

"Did you check for breathing?" Again, the coroner says, "No."

"So," the lawyer continues, "when you signed the death certificate, you had not taken any of the usual steps to make sure the man was dead, had you?"

The coroner, now tired of the browbeating, says, "Well, let me put it this way. The man's smashed brain was sitting in a police labeled jar on my desk, but for all I know, he could have been out there practicing law somewhere."

TTFN. If I hear any new ones, I'll let ya know.

[WmAx]

So much for a meaningful discussion, eh? Oh well, I guess it’s back to lawyer jokes.

I believe that pctower uses this forum primarily as practice for his debating skills. Most of the tactics I have noticed being employed by Phil can be found on how-to-debate lessons published online. I have read these tactics in the past. Seems to me that their main point is to not settle anything, but instead cause more confusion and to strike during this confusion. Does this sound familiar?

-Chris

[mtrycraft]

As far as my spelling, not really relevant to the issue(s) at hand.
-Chris

Interesting how now pc uses the spelling tactic to attack ones capability. He must have read too many e-stat messages attacking me on that issue. If that is all the ammo they have, let them enjoy their victory :D

[skeptic]

About the only thing that a DBT can usually tell you if it is carefully performed is whether it is likely or not that two things sound different or whether preference is likely the result of mere chance. It cannot tell you why they are different or whether or not that difference is significant.

I recently replaced my old CD player which bit the dust with another more reliable unit. I didn't need a DBT to know that they sound subtly but perceptably different. But the difference was due to minor differences in frequency response. This is nearly completely compensated for now through careful re-equalization of the system. I am convinced that once this is done, the old player would be indistinguishable from the new one on sound alone. IMO, this explains much of the perceived difference between comparable audio components.

Without careful electrical and acoustic measurements to isolate the variables and retest with further DBTs, it is virtually impossible for anyone to say that subtle audible differences between two different audio components are meaningful to an audiophile in an era when low cost equalizers are available to correct this "linear" distortion. That is undoubtedly why so many high end manufacturers don't like them. When a $400 preamplifier can be made to sound identical to a $4000 preamplifer, there isn't much point in buying the more expensive one. And don't think the high end guys are any less reluctant to tweak the response of their designs to be earcatching any more than the low end guys are.

[pctower]

I believe that pctower uses this forum primarily as practice for his debating skills. Most of the tactics I have noticed being employed by Phil can be found on how-to-debate lessons published online. I have read these tactics in the past. Seems to me that their main point is to not settle anything, but instead cause more confusion and to strike during this confusion. Does this sound familiar?

-Chris

If I had to depend on the debating skills I employ on this board to earn a living, I'd starve to death. This is pre-school in comparison to the league I play in.

[Monstrous Mike]

However, I note that you have totally ducked my request that you document your accusation that I am "wont" to change the subject. Simply expanding the scope of your accusation as you just did in your most recent post doesn't constitute documentation. It simply constitutes expanding the scope of your unsubstantiated allegation.
Isn't this just the expansion of an unsubstantiated segment constituting the lead in skit as documented on Saturday Night Live you recently tried to duck?

[mtrycraft]

Oh really! Would you care to post your resume so that we may all be enlightened as to how much more knowledgeable and accomplished you are than Professor Greenspun.

You must have read some of jneutron's posts about his exchane with Dr/professor Hawksford about Hawksford's paper 'Essexs Echo?' Yes, he is a professor at an English U.

[pctower]

Isn't this just the expansion of an unsubstantiated segment constituting the lead in skit as documented on Saturday Night Live you recently tried to duck?

Pardon me if I don't have a clue as to what you are talking about.

[pctower]

You must have read some of jneutron's posts about his exchane with Dr/professor Hawksford about Hawksford's paper 'Essexs Echo?' Yes, he is a professor at an English U.

I did. And so?

This is not a battle of the experts. This particular thread is a question I asked Mark about a statement from someone who is emersed in a scientific environment. That's all.

[Monstrous Mike]

Pardon me if I don't have a clue as to what you are talking about.

I was simply replying to your own baffling post in kind as shown below.


However, I note that you have totally ducked my request that you document your accusation that I am "wont" to change the subject. Simply expanding the scope of your accusation as you just did in your most recent post doesn't constitute documentation. It simply constitutes expanding the scope of your unsubstantiated allegation.
Hopefully, I got a chuckle out of the lurkers.

[markw]

Actually I think he's a pompus ass who likes to hear himself talk whether he knows what he's talking about or not. come to think about it, that's probably why you are so infatuated by him.

Why are you continuly asking me to comment on someone eles's opinon? This post has no bearing on anything I have ever said or implied.

Still looking to build that straw man, eh? If so, then sorry, I don't play that game. You'll just have to continue playing with yourself.

TTFN unless you want some more lawyer jokes.

[jneutron]

Actually I think he's a pompus ass who likes to hear himself talk whether he knows what he's talking about or not. come to think about it, that's probably why you are so infatuated by him.


Hmmm...

I'm still trying to find a post of Phil's that warrants your abusive statements..so far, I come up empty.

I'm also surprised that Chris has deemed your foul and vindictive language as appropriate for this forum.

I had hoped you would embrace the olive branch extended to you.. obviously, you have no desire for that.

Are you here for any reason other than trashing Phil?

John

[Richard Greene]

[QUOTE=pctower]Here's what one obvious "ignoramous" said on the subject:

"I've come to the conclusion that short term AB tests are pretty much worthless. I have one anecdotal story that illustrates this particularly well. "

RG
The logical comparison is between a Summer-long ABX test and Summer-long
subjective audition. Every day you could use the ABX comparator to choose one of the two pre-amps at random. After listening you'd write down your evaluation the listening experience as fair, good or excellent. At the end of the Summer you could see which
pre-amp got the best ratings when it was used. You probably wouldn't even have to check the scorecard by then (except maybe for cables, which many say all sound the same)
I assume days when you were in a bad mood and rated a component poorly for that reason would "even out" over several months and not affect the results.

More important -- who gets to borrow components for a whole Summer ?
I can just see the next time I stop in an audio store near closing on a Saturday and ask to borrow a component for an audition at home: "Since you're closed on Sunday, I'll bring it back first thing Monday morning ... oh, about two months from now."

The truth is I'd get to borrow a component for 1 1/2 days and during that short time I'll try a single-blind cable-swap A-B comparison if I can get a friend to help out. At least then
I know the excitement of a new component won't sway my purchase decision. There's no doubt in my mind that a subtle difference could be missed during a few hours of blind listening. However a subtle difference is rarely worth many Dollars to me. I'd rather invest in more CD's -- especially those with high quality sound ... or some new Sonex for early reflection points on my side walls. Or better yet, a new pointed stick to poke at those pesky WireNuts.

[markw]

I'll try to explain in as best I can, but if that's not good enough, well.


Hmmm...

I'm still trying to find a post of Phil's that warrants your abusive statements..so far, I come up empty.

Well, it kinda started out in the bi wire thread and went down hill from there when it was hijacked into a DBT based issue. I don't like people consistently putting words in my mouth and this seems to be a popular debating tactic for some.




I'm also surprised that Chris has deemed your foul and vindictive language as appropriate for this forum.

Who knows what they think? Personally, I only respond in kind.





I had hoped you would embrace the olive branch extended to you.. obviously, you have no desire for that.

You call this an olive branch? Sorry, I don't like getting whipped with an olive branch. Why not ask me if I stopped beating my wife, yes or no? From the confrontational tone of Phil's post it's pretty obvious to me that it's simply another excuse to start a debate about something I had no interest in debating.


Are you here for any reason other than trashing Phil?

John

Good question. Actually, I participate quite freely in most of the other forums. In fact, although they won't admit it, many of the views I express on them run quite parallel with others on this board whose views on cables I don't quite embrace. Imagine that.

Now, on the other hand, I don't see Phil participating on any other forum except here. And, his participation here seems intended more to disrail any seemingly well intentioned disccussion. His pre-emptive strike against any anti biwire information gleaned here in his first paragraph his first post in the bi-wire thread should pretty much make that clear.

Again, I'm sorry if you can't see what I'm saying, but I gotta be me.

I'm willing to live and let live, but this thread was, IMNSHO, started to simply start another debate about something I've never disputed. If it wasn't confrontational, then what is?

[pctower]

I'll try to explain in as best I can, but if that's not good enough, well.

Well, it kinda started out in the bi wire thread and went down hill from there when it was hijacked into a DBT based issue. I don't like people consistently putting words in my mouth and this seems to be a popular debating tactic for some.

Who knows what they think? Personally, I only respond in kind.

You call this an olive branch? Sorry, I don't like getting whipped with an olive branch. Why not ask me if I stopped beating my wife, yes or no? From the confrontational tone of Phil's post it's pretty obvious to me that it's simply another excuse to start a debate about something I had no interest in debating.

Good question. Actually, I participate quite freely in most of the other forums. In fact, although they won't admit it, many of the views I express on them run quite parallel with others on this board whose views on cables I don't quite embrace. Imagine that.

Now, on the other hand, I don't see Phil participating on any other forum except here. And, his participation here seems intended more to disrail any seemingly well intentioned disccussion. His pre-emptive strike against any anti biwire information gleaned here in his first paragraph his first post in the bi-wire thread should pretty much make that clear.

Again, I'm sorry if you can't see what I'm saying, but I gotta be me.

I'm willing to live and let live, but this thread was, IMNSHO, started to simply start another debate about something I've never disputed. If it wasn't confrontational, then what is?

You call this an olive branch? Sorry, I don't like getting whipped with an olive branch. Why not ask me if I stopped beating my wife, yes or no? From the confrontational tone of Phil's post it's pretty obvious to me that it's simply another excuse to start a debate about something I had no interest in debating.

I didn't mean to beat you with the branch. I told you I'd give you the last shot at me. I'm sorry you took it the way you did. I intended it as a true peace offering.

I do need to say that it wasn't my decision for you to start debating DBTs. If you thought I was improperly diverting the subject, you could have simply said so and then refused further discussion on the subject. I take full responsibility for the things I said and I'm not happy with how I behaved. But I can't be held responsible for you engaging in a debate that you didn't want to be involved with.

Now, on the other hand, I don't see Phil participating on any other forum except here.

I'm not sure why participation on only one board is a sin, but if you think it's important you might run a search over at AA (particularly if you're interested in recent posts at Prophead) under my Phil Tower moniker and under my earlier pctower moniker.

Again. I have no desire to continue what I view as childish, distateful behavior (I'm talking about mine - not yours). So again, I apologize for the inappropriate tone and substance of my posts.

[markw]

Deal? Perhaps we both could learn from these fiascos.

[jneutron]

I'll try to explain in as best I can, but if that's not good enough, well.
I thank you for the effort..that is all I can ask.


Personally, I only respond in kind.

I posted to you only because you continued to up the ante, so to speak, with personal insults. Had you simply responded in kind, I would have said nothing. As, I also am guilty of responding in kind (just ask JR and JC)


You call this an olive branch?

No. I was referring to Phils post, where he said this: ""It's easy to get caught up in them. I'm about ready to step back and catch my breath. So I'll toss out an olive branch here. I'm prepared to ratchet down the heat. Since I just took one parting shot at you, I'll give you the last word and then lets see if we can possibly convince at least one or two people that we are actually adults.


Sorry, I don't like getting whipped with an olive branch. Why not ask me if I stopped beating my wife, yes or no?
I make the assumption that your comment was meant for me in response to my post...you did not understand what I was referring to..no problem.


In fact, although they won't admit it, many of the views I express on them run quite parallel with others on this board whose views on cables I don't quite embrace. Imagine that.
Now, on the other hand, I don't see Phil participating on any other forum except here.

If you peruse prophead, you will find that PT has debated the "naysayer" side as much as he has the "yay" side here. And, many of his posts are intended to promote discussion, debate, as well as promoting self-assessment..I also find that some people clearly respond with the "party line" answers, without any semblence of thinking that maybe, they are missing something..


And, his participation here seems intended more to disrail any seemingly well intentioned disccussion. His pre-emptive strike against any anti biwire information gleaned here in his first paragraph his first post in the bi-wire thread should pretty much make that clear.

I come away with a different opinion. He definitely intends to promote discussion, but I don't see him doing that at anyones expense.


Again, I'm sorry if you can't see what I'm saying, but I gotta be me.

There is no reason to apologize for a difference of opinion. It'd be pretty damn boring if there were none.


I'm willing to live and let live, but this thread was, IMNSHO, started to simply start another debate about something I've never disputed. If it wasn't confrontational, then what is?

Ummm..my take was the start of a dialogue..but Phil would know..

Cheers, John

PS...mark, you posted about ten seconds before I did..

[pctower]

Deal? Perhaps we both could learn from these fiascos.

Amen.