Null Hypothesis

[RobotCzar]

I am taking a graduate-level stat course so I have had reasons for reviewing basic statistical concepts. In doing so, I now believe I can perhaps shed some light on the concept of the null hypothesis and some misunderstanding evident in posts in this forum.

1) First off-- the word "null" in null hypothesis has NOTHING to do with a null in audible differences. Lots of people are confused about this--including some people who are posting the results of blind testing. The null in the hypothesis refers to an assumed "no difference" in the mean of a sample (the test data) and the population from which the sample is taken (all humans that can hear in this case). The null assumption is necessary in order to make a judgment about the probability that your test results are due to chance.

2) The null hypothesis of an experiment is choosen as something you TRY to prove false. For this reason it is often chosen to be something that would be easy to show as false, but might be much harder to be shown as true in all cases. For example, if our null hypothesis is that "nobody can hear differences between two specific cables" we can easily find this to be false by finding ONLY ONE person who can. If the hypothesis were "some people can hear differences" it would be difficult to show this to be false as no matter how many people we test that hear no differences, we can suspect that someone else, not tested, can. Why do scientists always try to something is false rather than true? Well, demonstrating that something is false is usually easier as we only have to find one case were the the contention is false. The key point of this is that the way the null hypothesis is stated HAS NOTHING TO DO WITH WHAT THE EXPERIMENTER EXPECTS OR "WANTS" TO FIND IN THE EXPERIMENT. The conventions of stating the null in a particluar way relate to practical matters of logic.

Certain wire gurus (complete with Web sites), are demonstrating not only their lack of knowledge about valid testing, but also their muddle-headed thinking abilities, when they claim things like "you can't prove a null" or "scientists are trying to find a null in cable audibility" and so on. Rest assured the scientific and mathematical communities would be more than happy to point out how or why a peer-reviewed study is making mistakes in logic, theory, or experimental practice. Scientific reaseach studies are open to all kinds of criticism and carry much much more logical weight than all the 'theories" and "white papers" written by crackpots without scientific rigor.

[markw]

No, I'm not really a doctor but I like to keep it simple.

AFICT, the whole idea is to prove something. As far as testing for cable differences go, you either hear a difference or ypu don't. Simple as that. Now, as far as excluding wll the responses that essentially come out to "can't tell" is simply disallowing any response that proves there is no difference.

So, it would boil down to either you DO hear a difference or you DON'T hear a difference. Once that's accepted, then we get down to how many are correct. Of course, if you didn't hear a difference when a cable was changed, then you got it wrong, pure and simple. Likewise, if the cable wasn't changed and you still hear a difference, that's counted as a wrong also.

[okiemax]

I am taking a graduate-level stat course so I have had reasons for reviewing basic statistical concepts. In doing so, I now believe I can perhaps shed some light on the concept of the null hypothesis and some misunderstanding evident in posts in this forum.

1) First off-- the word "null" in null hypothesis has NOTHING to do with a null in audible differences. Lots of people are confused about this--including some people who are posting the results of blind testing. The null in the hypothesis refers to an assumed "no difference" in the mean of a sample (the test data) and the population from which the sample is taken (all humans that can hear in this case). The null assumption is necessary in order to make a judgment about the probability that your test results are due to chance.

2) The null hypothesis of an experiment is choosen as something you TRY to prove false. For this reason it is often chosen to be something that would be easy to show as false, but might be much harder to be shown as true in all cases. For example, if our null hypothesis is that "nobody can hear differences between two specific cables" we can easily find this to be false by finding ONLY ONE person who can. If the hypothesis were "some people can hear differences" it would be difficult to show this to be false as no matter how many people we test that hear no differences, we can suspect that someone else, not tested, can. Why do scientists always try to something is false rather than true? Well, demonstrating that something is false is usually easier as we only have to find one case were the the contention is false. The key point of this is that the way the null hypothesis is stated HAS NOTHING TO DO WITH WHAT THE EXPERIMENTER EXPECTS OR "WANTS" TO FIND IN THE EXPERIMENT. The conventions of stating the null in a particluar way relate to practical matters of logic.

Certain wire gurus (complete with Web sites), are demonstrating not only their lack of knowledge about valid testing, but also their muddle-headed thinking abilities, when they claim things like "you can't prove a null" or "scientists are trying to find a null in cable audibility" and so on. Rest assured the scientific and mathematical communities would be more than happy to point out how or why a peer-reviewed study is making mistakes in logic, theory, or experimental practice. Scientific reaseach studies are open to all kinds of criticism and carry much much more logical weight than all the 'theories" and "white papers" written by crackpots without scientific rigor.

GOOD FOR YOU! I hope you find the statistics course intellectually stimulating and learn useful information. Unless I'm missing something in your post, however, there may be a problem with your understanding of the null.

You said:" Certain wire gurus (complete with Web sites), are demonstrating not only their lack of knowledge about valid testing, but also their muddle-headed thinking abilities, when they claim things like "you can't prove a null" or "scientists are trying to find a null in cable audibility" and so on."

I can't comment on your remarks about wire gurus, because I don't know who you have in mind. However, you seem to be saying you can prove a null, which contradicts the following statement you made earlier:

"If the hypothesis were "some people can hear differences" it would be difficult to show this to be false as no matter how many people we test that hear no differences, we can suspect that someone else, not tested, can."

Here you seem to be saying you can't prove a null. I believe that would be the correct thing to say.

Good luck with the course!

[RGA]

I can't speak to what cable guru's claim. I think where the biggest break-down in all of the arguments over such inane issues as this is the statistical separation from the very goal of the test at the outset. I also have spoken to psychologists and PHD students in brain research who don't accept the null hypothesis nor would they accept double blind tests to prove or disprove cognitive brain function studies...which audio and any test of human subjects most certainly falls under - which isn't to say it should not be used either of course.

Though I certainly wouldn't spend any money on cables - so you can tell where I sit on the fence of cables. Though interestingly I did here a $100.00Mit cable once that sounded very different from the first set of cables...this was sighted and not level matched - he simply replaced the cable - both were probably 8foot sets. The MIT was quiter and rolled of the treble and muddied the bass. $100.00 and the sound was WORSE. Whatever MIT put in the box made it worse IMO. Anyone can DELIBERATELY make something like this sound different and it is in these companies' best interests to do so. Most will hear the difference and buy.

The fact that none of these companies gets positives in DBT's is odd IMO because if people can't hear the deliberate reverberation of a Rega Planet CD player(which is a deliberate attempt to alter the sound to make it different) then I wonder about the tests...then again perhaps that specific unit was not tested.

[RobotCzar]

I had hoped to make it clear that the word "null" and what you are testing for are NOT related. The concept of null in testing relates to the assumption of a null difference in that sample (test cases) mean and the mean of the whole population (humans who can hear in this case).

The other point I wanted to make was that the "null" hypothesis (it is just a name, not a test for a null) is stated in a way that makes it easy (or easier) to reject based on the data. (And not on what the experiment hopes to find evidence for.)

The target of post is people reading the misleading assumptions and opinions of unnamed cable gurus (who have web sites, offer up lots of technobabble on cables, and sometimes have the initials JR--hint hint).

No scientific test, especially statistical samples, can be said to prove anything. The study merely lends support or non-support to a hypothesis. Is the fact that nothing is "proved" significant? No, because what IS established is whether or not there is evidence supporting a hypothesis. So even though we can't say any individual test "proves" that cable differneces are inaudible (or are audible) we CAN make a very powerfual statement like "there is no evidence people can hear cable differences". This statement is sufficient to lead us to reject the usefulness of expensive cables (on the quality of our music reproduction). Anyone claiming that cables make "huge" or even subtle differences has the burden to produce some evidence that they do. Claims made in uncontrolled listening are useless as evidence as they don't isolate the source of the claimed differences (including listener bias and illusion).

[okiemax]

I had hoped to make it clear that the word "null" and what you are testing for are NOT related. The concept of null in testing relates to the assumption of a null difference in that sample (test cases) mean and the mean of the whole population (humans who can hear in this case).

The other point I wanted to make was that the "null" hypothesis (it is just a name, not a test for a null) is stated in a way that makes it easy (or easier) to reject based on the data. (And not on what the experiment hopes to find evidence for.)

The target of post is people reading the misleading assumptions and opinions of unnamed cable gurus (who have web sites, offer up lots of technobabble on cables, and sometimes have the initials JR--hint hint).

No scientific test, especially statistical samples, can be said to prove anything. The study merely lends support or non-support to a hypothesis. Is the fact that nothing is "proved" significant? No, because what IS established is whether or not there is evidence supporting a hypothesis. So even though we can't say any individual test "proves" that cable differneces are inaudible (or are audible) we CAN make a very powerfual statement like "there is no evidence people can hear cable differences". This statement is sufficient to lead us to reject the usefulness of expensive cables (on the quality of our music reproduction). Anyone claiming that cables make "huge" or even subtle differences has the burden to produce some evidence that they do. Claims made in uncontrolled listening are useless as evidence as they don't isolate the source of the claimed differences (including listener bias and illusion).

I get the gist of what you are saying, but I'm still not sure about your understanding of hypothesis testing. You said: "No scientific test, especially statistical samples, can be said to prove anything." I disagree, and will try to show why with a hypothetical example.

Suppose we are doing a controlled double-blind listening test on a $10 cable and a $100 cable with 20 subjects and 15 trials. The hypothesis and null hypothesis are stated as follows: hypothesis -- there is an audible difference in the two cables; null hypothesis -- there is no audible difference in the two cables. If one subject correctly identifies the cables enough times(e.g., 14 out of 15) to remove any doubt his score is a result of chance, the hypothesis is confirmed, even if the score of each of the remaining 19 subjects is no better than random. Because of the way the hypothesis was stated(i.e., existence of an audible difference), one listener is enough to prove it. If this isn't exactly what a researcher wants to find out, he can try to state the hypothesis in a different way.

It also would seem reasonable to suspect most people would not notice an audible difference in the $10 cable and the $100 cable, and that for them spending 10 times as much for no difference makes no sense. On the other hand, if an individual hears a difference, the worth of the more expensive cable to him would be for him to determine. His right to decide how to spend his money, however, does not depend on whether the decision is based on sighted or blinded listening. Nor is he obligated to scientifically verify his hearing claim or justify the expense before telling others about the purchase.

[RobotCzar]

I get the gist of what you are saying, but I'm still not sure about your understanding of hypothesis testing. You said: "No scientific test, especially statistical samples, can be said to prove anything." I disagree, and will try to show why with a hypothetical example.

Suppose we are doing a controlled double-blind listening test on a $10 cable and a $100 cable with 20 subjects and 15 trials. The hypothesis and null hypothesis are stated as follows: hypothesis -- there is an audible difference in the two cables; null hypothesis -- there is no audible difference in the two cables. If one subject correctly identifies the cables enough times(e.g., 14 out of 15) to remove any doubt his score is a result of chance, the hypothesis is confirmed, even if the score of each of the remaining 19 subjects is no better than random. Because of the way the hypothesis was stated(i.e., existence of an audible difference), one listener is enough to prove it. If this isn't exactly what a researcher wants to find out, he can try to state the hypothesis in a different way.

It also would seem reasonable to suspect most people would not notice an audible difference in the $10 cable and the $100 cable, and that for them spending 10 times as much for no difference makes no sense. On the other hand, if an individual hears a difference, the worth of the more expensive cable to him would be for him to determine. His right to decide how to spend his money, however, does not depend on whether the decision is based on sighted or blinded listening. Nor is he obligated to scientifically verify his hearing claim or justify the expense before telling others about the purchase.

No, tests that use samples (i.e., satistical tests) deal with probabilities. When you say that scoring 14 out of 15 "removes all doubt" you are incorrect. It is simply unlikely, not a certain proof that the person can hear a difference. Statistical tests make decisions to accept or reject hypothesis based on a convention of unlikelyness not certainty. There is a known probably of effor (type 1 or 2) in all statistical tests.

People are free to spend their money as they see fit--no matter how foolish the reasons

[mtrycraft]

The fact that none of these companies gets positives in DBT's is odd IMO because if people can't hear the deliberate reverberation of a Rega Planet CD player(which is a deliberate attempt to alter the sound to make it different) then I wonder about the tests...then again perhaps that specific unit was not tested.

Well, maybe one would get a positive result under DBT for just such a component IF the 'reverb' is great enough. Being under DBT is not a masking condition of differences. Actually, the senses are even sharper as you are not distracted by other sensory inputs.

One of my citations for JND, just noticable differences, is a peer paper for threshold detections. Conducted under forced choice DBT. It works. Just because it doesn't produce answers one wants or thinks it should is not an indication of DBts inability to deliver results.

[92135011]

o.o if you wanted to get technical you would need a control group
a group that listens to the same cable 2 times.
If you see a statistical difference then you know that there are problems.

but whatever, you can judge with your own ears.
However, for everything there is a limit.
Do you enjoy it to the extent to put out 500 bux for a pair of cables? Price controls all.

[Monstrous Mike]

RobotCzar:

Here is where I believe we are at and please correct me if I am wrong.

I personally hold a null hypothesis that there are no audible differences between a proper basic cable and the ones that are claimed to have "higher performance". This null cannot be proven true, or it would be extremely difficult to prove true.. However, it would be fairly easy to prove it false by having one properly contolled test done that shows that at least one person can consistently tell the difference.

And the very fact that there is not one single test in controlled conditions that proves this null hypothesis false is pretty strong support for it being true IMHO.

If anyone here feels so strongly that they have cables that make an audible difference, then I urge you to go to your local college's electrical engineering department and challenge some professor to do some proper testing on you and your cable. If the test comes out positive then you would be a hero to the audiophile community. Further you would have proven my null hypothesis false and now us scientists could get to work on finding out why your particular cable made a difference instead of chasing ghosts.

[kexodusc]

Certain wire gurus (complete with Web sites), are demonstrating not only their lack of knowledge about valid testing, but also their muddle-headed thinking abilities, when they claim things like "you can't prove a null" or "scientists are trying to find a null in cable audibility" and so on.

I certainly don't like the wire gurus or their websites, but I must come to their defense here...
If a wire gure claims that "you can't prove a null" they are telling the gospel truth. You can't. Not possible. And not proper statistics. You can reject the null, you cannot prove the null. The problem is thes goofy cable gurus aren't supporting their argument any by making such a claim.
Basically what they're saying here is "We're not going to prove to you there is an audible difference, instead we're just going to tell you that you cannot absolutely, with 100% confidence, prove that there IS NOT an audible difference."
As you've suggested, it is the seller's burden to provide proof, not the consumer's.

Great thread!

[skeptic]

Now you are gaining some insight into how real scientific knowledge is acquired. The statistical analysis of data is only one aspect of it.

The scientist who wants to test his hypothesis sets up the fairest test he can think of. He then conducts the test with enough participants and enough trials to be statistically significant. If he prequalifies all of the participants or segregates them into sub groups he tracks that too. The data is analyzed using statistical means to determine if the results are significantly different from the null meaning no statistical difference from random chance. He publishes his results including every aspect of his test procedure in a respected scientific journal which will be subject to review and discussion as well as repetition by his peers. If he finds the results statistically significant from the null, then his hypothesis is assumed to have validity until someone else comes along with a different test to challenge him. When that happens, there must be an investigation to determine why the first test got different results and plausible explanaitons convincing to the profession. When a body of knowledge is built up leading to the same conclusion over a long period, the hypothesis becomes the prevailing theory but is never immune to being challenged.

Now how do audiophiles come to their scientific conclusion about cables? They disconect one pair, try another, and proclaim one far superior to the other without ever having gone through even the most rudimentary analysis of what else might be different of if the comparison is even fair such as their old cables being corroded from years of sitting exposed to the air.

Can cable manufacturers do this? Do they know about this type of test? You can bet your last dollar that every single one of them does. Then why don't they do it? Why wouldn't they want to prove to scientists, engineers and audiophiles alike the superiority of their products? I can only assume the answer. And the answer is a) they know that they can't because their own electrical tests already has shown them that no usable difference exists and b) they don't have to because they already have a market that would only be diminished by the negative results of a test. Or maybe they have already conducted the tests and just don't want anyone to know the real results.

[Monstrous Mike]

The scientist who wants to test his hypothesis sets up the fairest test he can think of.

That's a good post skeptic. The response to it could only be something unscientific, anecdotal or illogical.

The fact is that there is even a standard that one can use to conduct proper subjective evaluations of audio signals. It is called "ITU-R Recommendation BS.1116". This standard was used to develop audio compression coding algorithms like MPeg-3. It is a real, documented, adopted, accepted and utilized method for subjectively evaluating audio signals.

All of the "shootouts" in hi fidelity magazines comparing cables and amps and CD players are basically just audio masterbation. The same applies to discussions on audio forums about in-home "trials" which supposedly show conclusive evidence of sonic differences between various audio products.

If you think about it, other than telling people what cable is correct one to use for the signal at hand, where to get it for a good price and how to hook it up, all the rest is simply fluff. Face it, these debates regarding cables and amps are not about science or truth or reality; they're about entertainment and that's about all.

If that's your bag, then I suggest you Cable Forum where your entertainment is protected from people like me and skeptic by higher beings.

[skeptic]

That is merely the first rung on a very tall ladder to arrive at the conclusion that swapping audio cables is a way to improve the performance of a sound system. Because even if you come to the conclusion that audible differences can exist, there's a lot more work to do to arrive at the recomendation for using them as a tool. The next step would be to understand what the differences are, what causes them, why they happen, and under what circumstances. And who could hear them. After that, the differences must be shown to be superior, not merely different. That's the science. Then comes the engineering. Next you must show that the method for obtaining these improvements, the swapping of cables is the most effective or cheapest way to achieve the beneficial results. Then there are questions which must be answered about how and when to use them, and which ones to use in a particular circumstance. That's a very long way from; they sound different so go out and buy them.

[okiemax]

Now you are gaining some insight into how real scientific knowledge is acquired. The statistical analysis of data is only one aspect of it.

The scientist who wants to test his hypothesis sets up the fairest test he can think of. He then conducts the test with enough participants and enough trials to be statistically significant. If he prequalifies all of the participants or segregates them into sub groups he tracks that too. The data is analyzed using statistical means to determine if the results are significantly different from the null meaning no statistical difference from random chance. He publishes his results including every aspect of his test procedure in a respected scientific journal which will be subject to review and discussion as well as repetition by his peers. If he finds the results statistically significant from the null, then his hypothesis is assumed to have validity until someone else comes along with a different test to challenge him. When that happens, there must be an investigation to determine why the first test got different results and plausible explanaitons convincing to the profession. When a body of knowledge is built up leading to the same conclusion over a long period, the hypothesis becomes the prevailing theory but is never immune to being challenged.

Now how do audiophiles come to their scientific conclusion about cables? They disconect one pair, try another, and proclaim one far superior to the other without ever having gone through even the most rudimentary analysis of what else might be different of if the comparison is even fair such as their old cables being corroded from years of sitting exposed to the air.

Can cable manufacturers do this? Do they know about this type of test? You can bet your last dollar that every single one of them does. Then why don't they do it? Why wouldn't they want to prove to scientists, engineers and audiophiles alike the superiority of their products? I can only assume the answer. And the answer is a) they know that they can't because their own electrical tests already has shown them that no usable difference exists and b) they don't have to because they already have a market that would only be diminished by the negative results of a test. Or maybe they have already conducted the tests and just don't want anyone to know the real results.

I don't know why cable makers don't do DBT's to prove the worth of their products, but my guess is somewhat different than yours. Suppose a cable maker finds through blinded testing that listeners can hear a difference between one of their expensive cables and lamp cord. What would be the point of publishing such a finding? Wouldn't prospective buyers already assume there was a difference? Wouldn't the firm's competitors in the cable business be laughing their butts off.? I can't think of any business where firms voluntarily do testing on their products in an attempt to make less money.

[mtrycraft]

I don't know why cable makers don't do DBT's to prove the worth of their products, but my guess is somewhat different than yours. Suppose a cable maker finds through blinded testing that listeners can hear a difference between one of their expensive cables and lamp cord. What would be the point of publishing such a finding? Wouldn't prospective buyers already assume there was a difference? Wouldn't the firm's competitors in the cable business be laughing their butts off.? I can't think of any business where firms voluntarily do testing on their products in an attempt to make less money.

Assuming is inconclusive. Knowing is definitive.
But, the real reason for their lack of DBT is the obvious, no differences.

[RGA]

Assuming is inconclusive. Knowing is definitive.
But, the real reason for their lack of DBT is the obvious, no differences.

Actually no it's not obvious...that statement requires proof.

[pctower]

Actually no it's not obvious...that statement requires proof.

Mytrycrafts and his followers don't need proof for their claims. They are "scientists" and they are above having to produce the kind of proof they demand of others. One might call them hypocritical, but one can't, because, after all, they're "scientists" and therefore beyond reproach.

They'll tell you, correctly so, that one can't prove a negative (such as "cables don't sound different"). But in the same breath they will declare the negative to be unqualifiedly true. When challenged they resort to sophistry and obfuscation. But they're "right" and you're "wrong", because, after all, they're "scientists".

[skeptic]

I don't know why cable makers don't do DBT's to prove the worth of their products, but my guess is somewhat different than yours. Suppose a cable maker finds through blinded testing that listeners can hear a difference between one of their expensive cables and lamp cord. What would be the point of publishing such a finding? Wouldn't prospective buyers already assume there was a difference? Wouldn't the firm's competitors in the cable business be laughing their butts off.? I can't think of any business where firms voluntarily do testing on their products in an attempt to make less money.

Wouldn't the other cable makers be laughing and ridiculing the ONE AND ONLY competitor who could PROVE his cables were better than lamp cord. On the contrary, If I ran that company, my add compaign would be, "I can prove you are getting something of real value for your money when you buy MY cable product. Nobody else can.)

"I can't think of any business where firms voluntarily do testing on their products in an attempt to make less money"

If you feel that your cable is superior, you test it to prove to the world that it really is and then you sell more and make more money. Why would you think that testing would result in making less money? Only one logical reason and that is if you test it and the results show it is no better than a much cheaper alternative, and this test becomes publically known, then there would be no reason for anyone to buy it.

Most companies test their products or send them out for testing for many different things for many different reasons. They also test their competitors products. It was funny. I once worked for a subsidiary of a large pharmaceutical company that manufactured surgical needles. They tested their own and their competitors needles endlessly. In fact, when I was there, seeing all of that industry's leadng company's products they kept buying for testing, I said that they were their competitors number one customer.

[okiemax]

Wouldn't the other cable makers be laughing and ridiculing the ONE AND ONLY competitor who could PROVE his cables were better than lamp cord. On the contrary, If I ran that company, my add compaign would be, "I can prove you are getting something of real value for your money when you buy MY cable product. Nobody else can.)

"I can't think of any business where firms voluntarily do testing on their products in an attempt to make less money"

If you feel that your cable is superior, you test it to prove to the world that it really is and then you sell more and make more money. Why would you think that testing would result in making less money? Only one logical reason and that is if you test it and the results show it is no better than a much cheaper alternative, and this test becomes publically known, then there would be no reason for anyone to buy it.

Most companies test their products or send them out for testing for many different things for many different reasons. They also test their competitors products. It was funny. I once worked for a subsidiary of a large pharmaceutical company that manufactured surgical needles. They tested their own and their competitors needles endlessly. In fact, when I was there, seeing all of that industry's leadng company's products they kept buying for testing, I said that they were their competitors number one customer.

I thought you were calling on cable manufacturers to demonstrate the worth of their products in blinded testing and to make the results of these studies public through advertising or other means Am I wrong? If not, can you give me specfic examples of manufacturers in other industries who have done blinded testing on their products, if not compelled to by law, and have made the results of these studies public. Perhaps there have been some, but I don't know if the practice is widespread, since no examples come to mind. What I am asking for is the names of companies and products, and how the results of the studies were made public. Only scientific studies are of interest.

[E-Stat]

I can only assume the answer. And the answer is a) they know that they can't because their own electrical tests already has shown them that no usable difference exists and b) they don't have to because they already have a market that would only be diminished by the negative results of a test. Or maybe they have already conducted the tests and just don't want anyone to know the real results.

or d) they are responsible businessmen making sound economic decisions.

Set aside cables for a minute. I can't think of a single audio manufacturer, be they low end or high end, that uses DBT or any other kind of statistical results in their marketing. Do you ever see an ad like this?

Double blind testing shows that four out of five dentists prefer Panasonkyo receivers.

It doesn't happen. Why? Well for one, properly run statistical testing is not inexpensive. Many audio companies, especially cable manufacturers, are small businesses. Most importantly, it is only the techie geeks who give a rat's ass as to the statistical outcome. Why waste precious marketing funds on tests that are judged irrelevant by your target market? You must have worked for the government at some time.

rw