Proof of placebo effect ?

[Monstrous Mike]

Question:

In the Tag McLaren tests, two participants scored 11 out of 15 correct on the cable tests. As they point out, the binomial distribution indicates a 5.9% (about 1 in 17) probability of scoring 11 or better by chance.

Why would one not conclude that in all liklihood those two individuals heard true differences in the cables?

Umm, because they hit that 1 in 17 chance thingy?

Surely you know that if you flip a coin long enough you are going to get ten heads in a row. Does this mean the coin is imbalanced? Hey, I hit 20 free throws in a row once. Does this mean I have a good shot? No, it means that some days, the sun even shines on a dog's ass.

You can't isolate test results to support your theories. You wouldn't let me do it and I won't let you do it.

[pctower]

Umm, because they hit that 1 in 17 chance thingy?

Surely you know that if you flip a coin long enough you are going to get ten heads in a row. Does this mean the coin is imbalanced? Hey, I hit 20 free throws in a row once. Does this mean I have a good shot? No, it means that some days, the sun even shines on a dog's ass.

You can't isolate test results to support your theories. You wouldn't let me do it and I won't let you do it.

I've already addressed the coin flip analogy, which is not appropriate. I'm surprised you don't see the difference. Or did you just think I would be too stupid not to see it.

That analogy assumes all listeners are identical as to hearing ability, experience and training (remember, your god, Dr. Toole, say training of the listeners is very important). Every flip of the coin is equivalent. But each listener is a unique "packet" and different from all the other "packets". In a coin toss I can pick any interval of 15 tosses I want in order to come up with the particular ratio out of 15 tosses I'm looking for. Can't do that with the individual "packets".

I don't claim the McLaren tests show anything reliable because of the two subjects that hit 11 or 15. However, if they came back 4 or 5 times and repeated that performance, only a few people such as yourself would find some weird way of ignoring those results.

[pctower]

BTW, you're the engineer. Can you answer this:

As I recall, there was a total of 12 listeners. As to the 10 who didn't achieve a level of accuracy of 10 out of 15 or better, the odds are their choices were due to chance and we might reasonable conclude they were unable to actually detect sonic differences.

As to the two who did achieve 11 out of 15, the McLaren site says the odds are 1 in 17 that the 11 correct choices out of 15 was due to chance for each on the two individuals. Obviously for each individual, that leaves a real, if not fairly remote, possibility his score was due to chance (as you suggested in your original post).

However, can you answer this? What are the odds that the result of two listeners, out of a total of 12, getting 11 out of 15 correct being due to chance? Seems to me, the odds would have to be far less than 1 out of 17.

I think my focus on the individuals is entirely appropriate. I don't know statistics, but I suspect it has something to do with distribution or something like that.

In my opinion, we should not be testing the mean or average ability of the entire group to pick correctly. I think what we should be attempting to do is determine if there are certain indivuduals who are actually hearing sonic differences. It's the "packet" thing I discussed above. I believe this is the appropriate focus, because as I have already said, each person's hearing ability, experience and training is different. So averages mean nothing. The proper question in my mind is whether there are certain people who due to experience, training or hearing ability are able to detect actual sonic differences where all possible factors other than actual sonic differences have been properly controlled out of the test.

What I have said should not be interpreted as claiming that these particular test results as to the two individuals are conclusive of anything. However, as I have already said, if these two individuals came back 4 or 5 more times and duplicated their initial scores, it seems to me that would be pretty persuasive evidence that these two individuals are detecting actual sonic differences.

In the words of Bill O'Reily: what say you?

[Monstrous Mike]

What I have said should not be interpreted as claiming that these particular test results as to the two individuals are conclusive of anything. However, as I have already said, if these two individuals came back 4 or 5 more times and duplicated their initial scores, it seems to me that would be pretty persuasive evidence that these two individuals are detecting actual sonic differences.

In the words of Bill O'Reily: what say you?

You are right. And let's say that these guys were better listeners for the sake or arguement. If they were able to repeat (e.g. in 4 or 5 more trials) an 11 out of 15 score then there may be something at work.

But I would still concerned if they consitently get 4 out of 15 wrong. How would one explain that? If there are cable differences I would think you would get it right every single time. I mean isn't there anybody who brought home an exotic cable and didn't hear a difference or are all these home audio enthusiasts noticing cable improvements at a 100% rate? It seems that way to me. This is odd indeed.

[Swerd]

The proper question in my mind is whether there are certain people who due to experience, training or hearing ability are able to detect actual sonic differences where all possible factors other than actual sonic differences have been properly controlled out of the test.

I agree that it is curious that those two scored that way. The Tag McLaren study didn’t address experience, training, hearing ability, or any other reason why listeners could or could not hear differences. It only asked the question “who can?” out of a small sized group of listeners and went through a typical statistical analysis of the numbers.

If those two individuals were retested again to see how reproducible their scores were, I would want to see additional control tests done. The ABX test as described by the Tag McLaren test did not include any controls:

The actual process of the test was to play 30 seconds of A followed by 30 seconds of B then 30 seconds of X, where X could be either A or B. Each time the piece of music was played it was announced as “This is A”, “This is B”, “This is X” etc. The participants in the test had a score card on which to record whether they thought X was A or B. The test was repeated 15 times.

I would do negative control tests, an A–A or B–B test, where the two sounds being compared were identical. Unlike the ABX tests, the listener would respond by saying whether or not he heard a difference. This would ask how often a listener reported, “I heard no difference” when there was none, or to state it conversely, how many false positive responses were made.

Similarly, I would do positive control tests where two genuinely different types of sound reproduction were compared, and again the listener would respond only by saying whether he heard a difference or not. This would ask how often a listener reported, “I did hear a difference” where one really existed, or how many false negatives were there. It would be useful to have an assortment of different reproduction modes that presented the listener with a graded series of comparisons from easy to difficult.

With those controls you could then compare the ABX results to something. How often listeners hear a difference for the positive control and how often they fail would be an important measure of the effectiveness of the experimental apparatus and of variability among individual listeners. A listener who answers yes to most of the positive controls and no to most of the negative controls could be considered as most reliable in the ABX test. Ideally, all listeners would hear a difference in the positive control and none would hear a difference in the negative control. However, it is possible to deviate from the ideal and still make useful conclusions, as long as suitable controls are included for each listener to determine his frequency of false negative and false positive responses.

The number of listeners who pass the control tests, regardless of their answers to the ABX test, might also be considered as a measure of validity for the whole experiment. The use of a positive control could help argue against potential criticism that subtle differences in sound due to different cables can only be heard after long-term listening by providing information of how often people can or cannot hear the differences in the positive control tests. It would be interesting to test golden-ears vs. tin-ears or cable believers vs. naysayers with such controls.

[mtrycraft]

Not long ago I read an interesting analysis of the number pi. As you ma know, the decimal numbers are random, no patterns has been established to hundreds of million decimal place.
Yet, there are two sequences of 10 succeeding odd or even digits in the 1st 1000 decimal places. No matter how small a chance havoing such a sequence, there are two within the 1st 1000 places. Just chance.
Same for this test.
Those two with high scores in one test didn't do as well in the next round.