Null Hypothesis

[RobotCzar]

I had hoped to make it clear that the word "null" and what you are testing for are NOT related. The concept of null in testing relates to the assumption of a null difference in that sample (test cases) mean and the mean of the whole population (humans who can hear in this case).

The other point I wanted to make was that the "null" hypothesis (it is just a name, not a test for a null) is stated in a way that makes it easy (or easier) to reject based on the data. (And not on what the experiment hopes to find evidence for.)

The target of post is people reading the misleading assumptions and opinions of unnamed cable gurus (who have web sites, offer up lots of technobabble on cables, and sometimes have the initials JR--hint hint).

No scientific test, especially statistical samples, can be said to prove anything. The study merely lends support or non-support to a hypothesis. Is the fact that nothing is "proved" significant? No, because what IS established is whether or not there is evidence supporting a hypothesis. So even though we can't say any individual test "proves" that cable differneces are inaudible (or are audible) we CAN make a very powerfual statement like "there is no evidence people can hear cable differences". This statement is sufficient to lead us to reject the usefulness of expensive cables (on the quality of our music reproduction). Anyone claiming that cables make "huge" or even subtle differences has the burden to produce some evidence that they do. Claims made in uncontrolled listening are useless as evidence as they don't isolate the source of the claimed differences (including listener bias and illusion).

[okiemax]

I had hoped to make it clear that the word "null" and what you are testing for are NOT related. The concept of null in testing relates to the assumption of a null difference in that sample (test cases) mean and the mean of the whole population (humans who can hear in this case).

The other point I wanted to make was that the "null" hypothesis (it is just a name, not a test for a null) is stated in a way that makes it easy (or easier) to reject based on the data. (And not on what the experiment hopes to find evidence for.)

The target of post is people reading the misleading assumptions and opinions of unnamed cable gurus (who have web sites, offer up lots of technobabble on cables, and sometimes have the initials JR--hint hint).

No scientific test, especially statistical samples, can be said to prove anything. The study merely lends support or non-support to a hypothesis. Is the fact that nothing is "proved" significant? No, because what IS established is whether or not there is evidence supporting a hypothesis. So even though we can't say any individual test "proves" that cable differneces are inaudible (or are audible) we CAN make a very powerfual statement like "there is no evidence people can hear cable differences". This statement is sufficient to lead us to reject the usefulness of expensive cables (on the quality of our music reproduction). Anyone claiming that cables make "huge" or even subtle differences has the burden to produce some evidence that they do. Claims made in uncontrolled listening are useless as evidence as they don't isolate the source of the claimed differences (including listener bias and illusion).

I get the gist of what you are saying, but I'm still not sure about your understanding of hypothesis testing. You said: "No scientific test, especially statistical samples, can be said to prove anything." I disagree, and will try to show why with a hypothetical example.

Suppose we are doing a controlled double-blind listening test on a $10 cable and a $100 cable with 20 subjects and 15 trials. The hypothesis and null hypothesis are stated as follows: hypothesis -- there is an audible difference in the two cables; null hypothesis -- there is no audible difference in the two cables. If one subject correctly identifies the cables enough times(e.g., 14 out of 15) to remove any doubt his score is a result of chance, the hypothesis is confirmed, even if the score of each of the remaining 19 subjects is no better than random. Because of the way the hypothesis was stated(i.e., existence of an audible difference), one listener is enough to prove it. If this isn't exactly what a researcher wants to find out, he can try to state the hypothesis in a different way.

It also would seem reasonable to suspect most people would not notice an audible difference in the $10 cable and the $100 cable, and that for them spending 10 times as much for no difference makes no sense. On the other hand, if an individual hears a difference, the worth of the more expensive cable to him would be for him to determine. His right to decide how to spend his money, however, does not depend on whether the decision is based on sighted or blinded listening. Nor is he obligated to scientifically verify his hearing claim or justify the expense before telling others about the purchase.

[RobotCzar]

I get the gist of what you are saying, but I'm still not sure about your understanding of hypothesis testing. You said: "No scientific test, especially statistical samples, can be said to prove anything." I disagree, and will try to show why with a hypothetical example.

Suppose we are doing a controlled double-blind listening test on a $10 cable and a $100 cable with 20 subjects and 15 trials. The hypothesis and null hypothesis are stated as follows: hypothesis -- there is an audible difference in the two cables; null hypothesis -- there is no audible difference in the two cables. If one subject correctly identifies the cables enough times(e.g., 14 out of 15) to remove any doubt his score is a result of chance, the hypothesis is confirmed, even if the score of each of the remaining 19 subjects is no better than random. Because of the way the hypothesis was stated(i.e., existence of an audible difference), one listener is enough to prove it. If this isn't exactly what a researcher wants to find out, he can try to state the hypothesis in a different way.

It also would seem reasonable to suspect most people would not notice an audible difference in the $10 cable and the $100 cable, and that for them spending 10 times as much for no difference makes no sense. On the other hand, if an individual hears a difference, the worth of the more expensive cable to him would be for him to determine. His right to decide how to spend his money, however, does not depend on whether the decision is based on sighted or blinded listening. Nor is he obligated to scientifically verify his hearing claim or justify the expense before telling others about the purchase.

No, tests that use samples (i.e., satistical tests) deal with probabilities. When you say that scoring 14 out of 15 "removes all doubt" you are incorrect. It is simply unlikely, not a certain proof that the person can hear a difference. Statistical tests make decisions to accept or reject hypothesis based on a convention of unlikelyness not certainty. There is a known probably of effor (type 1 or 2) in all statistical tests.

People are free to spend their money as they see fit--no matter how foolish the reasons

[okiemax]

No, tests that use samples (i.e., satistical tests) deal with probabilities. When you say that scoring 14 out of 15 "removes all doubt" you are incorrect. It is simply unlikely, not a certain proof that the person can hear a difference. Statistical tests make decisions to accept or reject hypothesis based on a convention of unlikelyness not certainty. There is a known probably of effor (type 1 or 2) in all statistical tests.

People are free to spend their money as they see fit--no matter how foolish the reasons

My post was in response to your statement that "No scientific test, especially statistical samples, can be said to prove anything." If the example of 1 of 20 subjects scoring 14 out of 15 isn't convincing , make it all 20 subjects scoring a perfect 20 out of 20. And make the sample of 20 subjects scientifically drawn. And get the same results with many additional scientific samples. Wouldn't that show that a scientific test with statistical sampling proved something?

[skeptic]

Not necessarily. Statistical analysis will give you a degree or interval of confidence which only tells you the probability that the results were or were not the effect of random chance. No statistical average is 100 percent certain. For instance, if you were to flip a coin 10,000 times, there is a very high probability that somewhere along the way, you would flip heads 15 out of 16 tries. If you were unlucky enought to randomly pick that sample to look at and draw the conclusion you did, it would be the wrong conclusion.

[okiemax]

Not necessarily. Statistical analysis will give you a degree or interval of confidence which only tells you the probability that the results were or were not the effect of random chance. No statistical average is 100 percent certain. For instance, if you were to flip a coin 10,000 times, there is a very high probability that somewhere along the way, you would flip heads 15 out of 16 tries. If you were unlucky enought to randomly pick that sample to look at and draw the conclusion you did, it would be the wrong conclusion.

What are you taking issue with? I was talking about perfect scores not just with one sample, but with multiple samples. Anyway, this is getting away from my original point which was a person can in a controlled listening test proove two cables are audibly different to him if he can correctly identify the cables a sufficient number of times to remove any possibility that his performance is a result of chance. Thus, if one person has demonstrated beyond a doubt that he can hear a difference, an audible difference in the cables exists.

[RobotCzar]

What we disagree with is the notion that you can remove "remove ANY possibility of performance due to chance". That is precisely what cannot be done, and is one reason that no study can "proove" a hypothesis.

Obviously, we humans can accept that very unlikely events offer some evidence that a hypothesis is true or false. That is the whole basis of statistical testing. We accept evidence that is not 100% proven for every instance all the time. We would be likely to say the a person correctly identifying 18 of 20 tests is evidence that that person can hear a difference, and it is.

Why do you not accept that repeated failure of people to do this is evidence that they cannot hear a difference?