Golden Ears Fail Again

[RobotCzar]

I see a problem with the experiment. Because your mood may change from one day to the next, your choices can be more a result of variations in how you feel than differences in the cables. This could result in the test having a negative bias. Given the method you will use, I don't know a way around the problem. It should not, however, in itself invalidate positive results.

Good luck!

One might also assume that a mood might have a positive bias.

So you (mystic) hypothesize that a person's mood can affect their ability to distinguish cable differences. I agree, even the time of day might. The fact that these variables can influence what you "hear" is in itself an indictment of uncontrolled listening tests. The advice to "just listen" is really bad advice. One must make some effort to even out all the subjective factors that might be affecting one's perceptions.

Your advice about random selection is excellent.

[mystic]

One might also assume that a mood might have a positive bias.

So you (mystic) hypothesize that a person's mood can affect their ability to distinguish cable differences. I agree, even the time of day might. The fact that these variables can influence what you "hear" is in itself an indictment of uncontrolled listening tests. The advice to "just listen" is really bad advice. One must make some effort to even out all the subjective factors that might be affecting one's perceptions.

Your advice about random selection is excellent.

I think in the proposed experiment the influence of mood can be only a negative bias. I'm in a bad mood, so I identify A (the cable preferred in sighted listening) as B. Or I'm in a good mood, so identify B as A . Misidentification obviously lowers the positive scores. I can't imagine how mood bias would work the other way(i.e., raise the scores). But you may have had something entirely different in mind when you said " one might also assume that a mood might have a positive bias."

I don't rule out that attitudes, beliefs, and expectations can influence what listeners think they hear or do not hear, be it sighted or blinded listening. But I'm not convinced that ABE's are the strong and pervasive influence some seem to think. If someone suggests I can be convinced through the power of suggestion that horse pee is beer, I'm going to say -- Hey, maybe you but not me! I believe things usually are as the senses indicate.

[Toga]

haha @ horse beer!

[RobotCzar]

"Misidentification obviously lowers the positive scores." Actually, misidentification lowers correct scores as that is the definition of misidentification. We are concerned about how a person's mood effects their peformance. Why is it hard to see that perhaps mood can make scored go up or down? You seem to imply that there are only two moods bad and good. Is there a state of neither bad nor good mood?

Well, if a "negative bias" would decrease the number correct of a subject (relative to some normal conditions--what they would score without the bias), then a "positive bias" would increase one's score relative to no bias at all. For example, if I really could distinguish A from B, then I might do better when I feel happy or am "pumped". Sometimes people say that they are "in the zone".

You to need to stretch your imagination a bit as one could demonstrate that they can hear differences--and have all their guesses be incorrect. If you did much worse than chance, you would be demonstrating that you can hear a difference, you are just misidentifying the choices. This does happen.

My main point was that if a mood can effect the outcome of a listening test, then one's expectations certainly could (as in a sighted test).

[mystic]

"Misidentification obviously lowers the positive scores." Actually, misidentification lowers correct scores as that is the definition of misidentification. We are concerned about how a person's mood effects their peformance. Why is it hard to see that perhaps mood can make scored go up or down? You seem to imply that there are only two moods bad and good. Is there a state of neither bad nor good mood?

Well, if a "negative bias" would decrease the number correct of a subject (relative to some normal conditions--what they would score without the bias), then a "positive bias" would increase one's score relative to no bias at all. For example, if I really could distinguish A from B, then I might do better when I feel happy or am "pumped". Sometimes people say that they are "in the zone".

You to need to stretch your imagination a bit as one could demonstrate that they can hear differences--and have all their guesses be incorrect. If you did much worse than chance, you would be demonstrating that you can hear a difference, you are just misidentifying the choices. This does happen.

My main point was that if a mood can effect the outcome of a listening test, then one's expectations certainly could (as in a sighted test).

"Positive" and "correct" in the context of the test under discussion mean the same thing to me. After the test, we count the number of times the subject identified the cable as positive (or correct)scores and the number of times he misidentified the cable as negative (or incorrect) scores. Perhaps I'm missing your point.

I didn't mean to imply a person is either in a good mood or a bad mood with no in between. I don't know about everyone, but I'm in between in varying degrees most of the time. However, when not in a good mood I don't like listening to music. Thankfully, music is enjoyable at other times, and can add to feeling good.

Please consider the following hypothetical example of moods hurting performance: The subject in the 13 day blinded test might say " I correctly identified Cable A (the one he preferred in sighted listening) 5 days out of 7 days, and probably would have done better, but wasn't in a good mood those two days so maybe the cable didn't sound good to me for that reason. And I identified Cable B on 5 of 6 days, the one wrong being a day when I felt particularily good."

You gave an example of how you thought a positive mood bias might enhance a person's score. "For example, if I really could distinguish A from B, then I might do better when I feel happy or am "pumped". Sometimes people say that they are "in the zone." If listeners need to be that way to do their best in blinded testing, I see it as a reason to question the validity of the testing. How can you assure that the people who need to be "pumped" will be during the testing?

I continue to wonder whether there is something inconsistent about assuming attitudes, beliefs, and expectations can affect a listener's perceptions while at the same time assuming the act of blinded testing cannot affect his listening performance. If the mind and the way the mind and ears work together can be influenced by so many things, why is the testing itself exempt?

You don't have to tell me about "have all their guesses be incorrect." I have had first-hand experience as a student in doing worse than random on true/false exams. If only I knew then what I know now, I would have tried to convince my instructors I knew the answers but just misidentified the choices. I know this is not the kind of wrong beyond random you are talking about, but it reminded me of those exams.

Reflecting on school experiences, I also recall futile efforts at trying to beat the odds on exam questions. For example, being pretty sure the first three answers on an exam were false, I might guess the fourth should be true unless I was certain it was false. Listener's uncertainty or insecurity about failing could cause "trying to beat the odds" in blinded testing, a distraction that I suspect would hurt performance.

You said " My main point was that if a mood can effect the outcome of a listening test, then one's expectations certainly could (as in a sighted test)." I agree, one's expectations could affect the outcome of a sighted test, but I don't believe they necessarily would. Also, I think expectations could work either for or against hearing differences.

[RobotCzar]

"Positive" and "correct" in the context of the test under discussion mean the same thing to me. After the test, we count the number of times the subject identified the cable as positive (or correct)scores and the number of times he misidentified the cable as negative (or incorrect) scores. Perhaps I'm missing your point..

I don't see what "positive" has to do with "correct". Postive (or good) relates to a mood (I thought from your comments). To my thinking your are confusing performance (%correct) with mood. Mood can affect performance: bad mood can decrease it, good mood can improve it (in comparison to "no mood" or neutral mood)--but they are different concepts.

Please consider the following hypothetical example of moods hurting performance: The subject in the 13 day blinded test might say " I correctly identified Cable A (the one he preferred in sighted listening) 5 days out of 7 days, and probably would have done better, but wasn't in a good mood those two days so maybe the cable didn't sound good to me for that reason. And I identified Cable B on 5 of 6 days, the one wrong being a day when I felt particularily good.".

Well, I would NOT let the subject explain what happended, I would let the data do so. If you want him to self-report his mood, fine. But his explanation of mood to performance is worthless. He shouldn't even know how he did on days he felt bad, because he is blinded. The test should merely look for correlations between self reported mood and performance. As an aside, the subejct could have said "I identified better on days when I was in a good mood, I did a bit worse on days I felt neither good or bad, and I did even worse on days I felt bad". (His explanations, however, carry no weight, the data will show if there is a correlation between mood and performance or not.)

You gave an example of how you thought a positive mood bias might enhance a person's score. "For example, if I really could distinguish A from B, then I might do better when I feel happy or am "pumped". Sometimes people say that they are "in the zone." If listeners need to be that way to do their best in blinded testing, I see it as a reason to question the validity of the testing. How can you assure that the people who need to be "pumped" will be during the testing?

I see no reason to question the testing procedure because it demonstrates that factors like mood (or time of day, or whether the subject just ate, or whether the subject took asprin, etc. etc.) affect performance. A host of factors can affect performance which is why some effort must be made to control them. The factor of expectation (or simple bias) can be controlled by blinding. Why do you think that procedure needs to be tested or checked or whatever? If you don't know which is which, you bias cannot affect your performance--it is that simple. Doesn't that make sense?

I continue to wonder whether there is something inconsistent about assuming attitudes, beliefs, and expectations can affect a listener's perceptions while at the same time assuming the act of blinded testing cannot affect his listening performance. If the mind and the way the mind and ears work together can be influenced by so many things, why is the testing itself exempt?

Blinding in a test does not mean cutting off eyesight. A typical audio ABX test lets people see just fine. Blinding in this context means removing the knowledge of which item test is which. You need to find out more, you are in no position to question testing procedure because you don't know what it is. What good is a test that tests if you can tell two things apart if you already know which is which. I wouldn't accept that as a test, would you?

Reflecting on school experiences, I also recall futile efforts at trying to beat the odds on exam questions. For example, being pretty sure the first three answers on an exam were false, I might guess the fourth should be true unless I was certain it was false. Listener's uncertainty or insecurity about failing could cause "trying to beat the odds" in blinded testing, a distraction that I suspect would hurt performance.

As I said, you don't know enough at this point. Test subjects do not know how they are doing on the test until it is over. That cannot try to "beat the odds" as you suggest, because they generally don't know their score.

You said " My main point was that if a mood can effect the outcome of a listening test, then one's expectations certainly could (as in a sighted test)." I agree, one's expectations could affect the outcome of a sighted test, but I don't believe they necessarily would. Also, I think expectations could work either for or against hearing differences.

Right, one's expectations may or may not affect the result a test. The only way to be sure they are not affecting the result is to blind the subject (remove knowledge of which component is which). You have just explained why blinding is required in a valid listening test.

[mystic]

I don't see what "positive" has to do with "correct". Postive (or good) relates to a mood (I thought from your comments). To my thinking your are confusing performance (%correct) with mood. Mood can affect performance: bad mood can decrease it, good mood can improve it (in comparison to "no mood" or neutral mood)--but they are different concepts.


Well, I would NOT let the subject explain what happended, I would let the data do so. If you want him to self-report his mood, fine. But his explanation of mood to performance is worthless. He shouldn't even know how he did on days he felt bad, because he is blinded. The test should merely look for correlations between self reported mood and performance. As an aside, the subejct could have said "I identified better on days when I was in a good mood, I did a bit worse on days I felt neither good or bad, and I did even worse on days I felt bad". (His explanations, however, carry no weight, the data will show if there is a correlation between mood and performance or not.)


I see no reason to question the testing procedure because it demonstrates that factors like mood (or time of day, or whether the subject just ate, or whether the subject took asprin, etc. etc.) affect performance. A host of factors can affect performance which is why some effort must be made to control them. The factor of expectation (or simple bias) can be controlled by blinding. Why do you think that procedure needs to be tested or checked or whatever? If you don't know which is which, you bias cannot affect your performance--it is that simple. Doesn't that make sense?


Blinding in a test does not mean cutting off eyesight. A typical audio ABX test lets people see just fine. Blinding in this context means removing the knowledge of which item test is which. You need to find out more, you are in no position to question testing procedure because you don't know what it is. What good is a test that tests if you can tell two things apart if you already know which is which. I wouldn't accept that as a test, would you?

As I said, you don't know enough at this point. Test subjects do not know how they are doing on the test until it is over. That cannot try to "beat the odds" as you suggest, because they generally don't know their score.


Right, one's expectations may or may not affect the result a test. The only way to be sure they are not affecting the result is to blind the subject (remove knowledge of which component is which). You have just explained why blinding is required in a valid listening test.

You accept results of blinded listening tests as truth. You do not accept results of sighted listening tests as truth. I get the impression you also don't believe a claim can be true unless it has been verified through blinded testing, but I may be misunderstanding you. You have confidence in the state of blinded testing in audio, believing its practice has always revealed the truth.

I accept results of objective tests as truth. Blinded listening tests do seem objective. But all this talk about how easily the mind is influenced makes me wonder about the influence of the testing itself. I'm not in a position where I have to assume the testing is objective for work or any other practical reason. So for me the reasonable conclusion is no conclusion.

[RobotCzar]

Let's suppose you meet a person who claims to be able to tell Coke from Pepsi. Knowing that human perception is easily misled, you ask him to demonstrate that he can. He agrees and you pour the colas into cups and set up a test. You attempt to hide which is which from him and he, at first, objects but eventually agrees. You mix up the drink so he doesn't know which is which (but it is written under a piece of tape on each cup. He driinks each, taking a long as he likes, and being able to take as big a sip as he wishes. You ask if he can tell the difference and he says "yes, for sure".

The test is completed and you add up his score and find that his correct number of responses is exactly what you would expect by chance from someone who could not tell the difference between the colas.

You tell him the results. He says that there is something wrong with the test, because he knows that he can tell the difference. He says that it is unfair to not let him know which is which before hand and suggests that the pressure of the test affected his ability to do the test. You test him again, this time he is not so cocky. Again he fails to demonstrate he can tell the difference.

He still claims he can. He says your test is flawed because he needs to know which is which to tell the difference (I'll bet that helps a lot). He says that you need to test your test because it does not reveal the truth. Why would he lie about this, he asks? You say, because you are fooling yourself--your perception of taste is being affected by your expectations. You point out this is a well-known scientific fact. He says you are a pin head and that all you care about is numbers, not reality. Besides, he loves Coke and you are spoiling his fun.

You meet another guy. He says he can easily tell Coke from Pepsi. You say "That's nice, but I can't believe you until you demonstrate that you can because you tested somebody and found out they couldn't. He says, "well that was him, I can do it," You test him and he cannot distinguish Coke from Pepsi. He calls you a naysayer and says: "you think everything tastes the same because you can't tell the difference." You say: "neither can you" and until somebody does, I am not sure anybody can. I know one thing for SURE, just because people SAY they can doesn't mean they really can.

This is a rough approximation of what has happened with blind listening tests of amps and cables. Except the situation is much worse for these audio compoents, Much time has gone by and nobody has demonstrated the ability to distinguish these type of components. In addition, scientific test has established that people's hearing sense cannot distinguish the differences in sound at the levels measured from audio components.

The bottom line is what you consider logical or fair is not what I do, nor, I claim--what a scientific opinion would be.