Testing and the Scientific Method

[pctower]

Some different perspectives:

http://www.rationalmagic.com/RMS/rms-jamacrit.html

http://faculty.ncwc.edu/toconnor/daubert.htm

http://www.uww.edu/Lettsc/core/STS/m...ocs/method.htm

http://koning.ecsu.ctstateu.edu/Plan...n/scimeth.html

http://www.epa.gov/maia/html/scientific.html

http://www.nafi.org/archives/archive0013.htm

http://home.xnet.com/~blatura/skep_1.html#1.1

[Pat D]

Some different perspectives:

http://www.rationalmagic.com/RMS/rms-jamacrit.html

http://faculty.ncwc.edu/toconnor/daubert.htm

http://www.uww.edu/Lettsc/core/STS/m...ocs/method.htm

http://koning.ecsu.ctstateu.edu/Plan...n/scimeth.html

http://www.epa.gov/maia/html/scientific.html

http://www.nafi.org/archives/archive0013.htm

http://home.xnet.com/~blatura/skep_1.html#1.1

Well, now, those are very interesting, though not at a very high level.

Let's look at the "dreadful secret" little secret about falsification, as Philip Kitcher calls it. One cannot verify a theory, as Popper said, but then one can't really falsify it, either. Suppose we find a phenomenon that does not obey Newton's law of gravity--does that disprove them? Well, no, because one can always hypothesize some other forces were at work that caused the phenomenon not to behave as predicted by the theory. As Kitcher points out, "historians philosophers of science have been trying to let this particular cat out of the bag" for decades. (See Philip Kitcher, Abusing Science, The Case Against Creationism, MIT Press, 1984, p. 42.

And, as Pierre Duhem pointed out, rather than being tested alone, "Hypotheses are tested in bundles." (Kitcher, op. cit., p. 44)

[Monstrous Mike]

Here is a quote from: http://home.xnet.com/%7Eblatura/skep_1.html#1.1

"On the other hand the theory that "there is an invisible snorg reading this over your shoulder" is not falsifiable. There is no experiment or possible evidence that could prove that invisible snorgs do not exist. So the Snorg Hypothesis is not scientific. On the other hand, the "Negative Snorg Hypothesis" (that they do not exist) is scientific. You can disprove it by catching one. Similar arguments apply to yetis, UFOs and the Loch Ness Monster. See also question 5.2 on the age of the Universe."

I would have to say that audio cables fit the Snorg Hypothesis quite well. It is interesting that it appears the negative of this hypothesis is actually the scientific one, i.e. cables do not sound different. So it looks like you have to catch one (i.e. show that cables can sound different) to disprove this Negative Snorg Hypothesis that I for one hold onto as the front-running candidate for the Truth.

[okiemax]

Here is a quote from: http://home.xnet.com/%7Eblatura/skep_1.html#1.1

"On the other hand the theory that "there is an invisible snorg reading this over your shoulder" is not falsifiable. There is no experiment or possible evidence that could prove that invisible snorgs do not exist. So the Snorg Hypothesis is not scientific. On the other hand, the "Negative Snorg Hypothesis" (that they do not exist) is scientific. You can disprove it by catching one. Similar arguments apply to yetis, UFOs and the Loch Ness Monster. See also question 5.2 on the age of the Universe."

I would have to say that audio cables fit the Snorg Hypothesis quite well. It is interesting that it appears the negative of this hypothesis is actually the scientific one, i.e. cables do not sound different. So it looks like you have to catch one (i.e. show that cables can sound different) to disprove this Negative Snorg Hypothesis that I for one hold onto as the front-running candidate for the Truth.

The kind of hypothesis we are discussing has to be statistically testable if a scientific experiment is to be done. A "cables sound different" hypothesis might be tested, but unless I'm too sleepy to think straight, hypotheses regarding the existence of invisible Snorgs, UFO's, and the Loch Ness Monster can't be tested. Why attempt to discredit testable claims by association with untestable absurdities?

I take back what I said about the existence of invisible Snorgs being untestable. I just experimented by glancing over my left shoulder to see if one was there. I did it 15 times to assure statistical significance, and never saw the Snorg once. If that doesn't proove an invisible Snorg was there, I don't know what does.

[Monstrous Mike]

The kind of hypothesis we are discussing has to be statistically testable if a scientific experiment is to be done. A "cables sound different" hypothesis might be tested, but unless I'm too sleepy to think straight, hypotheses regarding the existence of invisible Snorgs, UFO's, and the Loch Ness Monster can't be tested. Why attempt to discredit testable claims by association with untestable absurdities?

I take back what I said about the existence of invisible Snorgs being untestable. I just experimented by glancing over my left shoulder to see if one was there. I did it 15 times to assure statistical significance, and never saw the Snorg once. If that doesn't proove an invisible Snorg was there, I don't know what does.

I honestly don't understand your reply.

That author was saying that null hypotheses for Snorgs, UFOs, Bigfoot, etc. can be proven to be false. All you have to do is catch a Snorg or a Bigfoot or shoot down an alien craft. Bingo, the null hypothesis is falsified and these things do exist. We can go from there to find out details like where they came from, how long they've been around, why they were so hard to catch, etc.

I feel the null hypothesis for cables fits right into this line of thinking. We need to show conclusively that cables do sound different and then we can go about discovering why they sound different, in what circumstances they sound different, how different they sound to different people and how to improve the difference so it is enjoyed more.

As of today, cable differences are like the invisible Snorg. Some people claim to have seen him or felt him or are sure he has been in the room with them, but nobody can prove it.

[pctower]

I honestly don't understand your reply.

That author was saying that null hypotheses for Snorgs, UFOs, Bigfoot, etc. can be proven to be false. All you have to do is catch a Snorg or a Bigfoot or shoot down an alien craft. Bingo, the null hypothesis is falsified and these things do exist. We can go from there to find out details like where they came from, how long they've been around, why they were so hard to catch, etc.

I feel the null hypothesis for cables fits right into this line of thinking. We need to show conclusively that cables do sound different and then we can go about discovering why they sound different, in what circumstances they sound different, how different they sound to different people and how to improve the difference so it is enjoyed more.

As of today, cable differences are like the invisible Snorg. Some people claim to have seen him or felt him or are sure he has been in the room with them, but nobody can prove it.

"As of today, cable differences are like the invisible Snorg. Some people claim to have seen him or felt him or are sure he has been in the room with them, but nobody can prove it."

That's really your subjective opinion. The personal experience of most educated, rational people and what they know about science leads them to easily conclude that it is extremely unlikely that invisible Snorgs exists. The null hypothisis technically stated is that they don't exist.

The null hypothethesis technically stated for cable differences is they don't exist. I assume that your personal experience and what you know scientifically leads you to conclude that it is extremely unlikely that differences exist. Other educated, rational people (including a number of scientists and engineers who have really paid attention to audio cables and tried some or many in their systems) believe differently. For them their personal experience and what they know about science would in all likelihood lead them to agree with you as to the non-existence of invisible Snorgs, yet disagree with you about cables.

These people have no scientific proof that cable differences exist just as such evidence is lacking for Snorgs. But in those situations (such as Snorgs and audio cables) virtually no valid control testing has been done, different people based on their own personal experiences may come to different conclusions. In those areas such as cables where there is room (in my opinion) for rational people to differ in their "guesses" as to which is really true (cable differences either exist or they don't), I find it unfortunate that many seem to "look down" on those who have reached a difference conclusion.

The amount of energy expended by people who discuss audio cables in making people who disagree with them wrong, silly, incompentent and inhuman is rather amazing.

[Chuck]

I honestly don't understand your reply.
As of today, cable differences are like the invisible Snorg. Some people claim to have seen him or felt him or are sure he has been in the room with them, but nobody can prove it.

Hi Mike,

This is my first visit to AR in six months or more; What I can't understand is why so many people are wasting time arguing with other people who think they have invisible Snorg's reading over their shoulder. Rational men don't believe in Snorg's, so you know going in that you will be arguing with people who are irrational, and apparently delusional. Why not just treat them the same way you do the other groups who imagin things (like having been abducted by UFO's). I think it is highly discriminatory to spend all your time picking on the people who believe in the Snorg. The world is full of people who have delusions, so why waste months and years picking on the Snorgophiles? That just doesn't seem fair. What makes Snorgs so special?

Guess I’ll stop back in another six months or so, and see if anyone has learned anything. Seems that the last six months passed without any progress, since absolutely nothing has changed. What a complete waste of time:

[okiemax]

Here is a quote from: http://home.xnet.com/%7Eblatura/skep_1.html#1.1

"On the other hand the theory that "there is an invisible snorg reading this over your shoulder" is not falsifiable. There is no experiment or possible evidence that could prove that invisible snorgs do not exist. So the Snorg Hypothesis is not scientific. On the other hand, the "Negative Snorg Hypothesis" (that they do not exist) is scientific. You can disprove it by catching one. Similar arguments apply to yetis, UFOs and the Loch Ness Monster. See also question 5.2 on the age of the Universe."

I would have to say that audio cables fit the Snorg Hypothesis quite well. It is interesting that it appears the negative of this hypothesis is actually the scientific one, i.e. cables do not sound different. So it looks like you have to catch one (i.e. show that cables can sound different) to disprove this Negative Snorg Hypothesis that I for one hold onto as the front-running candidate for the Truth.

We already know you can't use hypothesis testing to prove something doesn't exist. The Snorg hypothesis is just the use of an absurdity to help explain why a negative can't be proved. I don't see anything new here. It looks like you are again trying to discredit claims about cables by comparing them to other claims that are patently absurd.

Labeling the hypothesis as "not scientific" because it can't be disproved, and labeling the null hypothesis as "scientific" because it can be disproved adds nothing to what we already know, and can be misleading. These labels might imply that the null hypothesis "cables sound the same" is somehow better than the hypothesis "cables sound different," which makes no sense.

I may be misinterpreting your remarks, but I wonder whether you are trying to turn things upside down and use "cables sound the same" as the hypothesis instead of the null hypothesis. If so, I don't think you can do that.

I'm sorry, Mike. I had intended this as a reply to you last post on this thread, but put it in the wrong place.

[Pat D]

Here is a quote from: http://home.xnet.com/%7Eblatura/skep_1.html#1.1

"On the other hand the theory that "there is an invisible snorg reading this over your shoulder" is not falsifiable. There is no experiment or possible evidence that could prove that invisible snorgs do not exist. So the Snorg Hypothesis is not scientific. On the other hand, the "Negative Snorg Hypothesis" (that they do not exist) is scientific. You can disprove it by catching one. Similar arguments apply to yetis, UFOs and the Loch Ness Monster. See also question 5.2 on the age of the Universe."

I would have to say that audio cables fit the Snorg Hypothesis quite well. It is interesting that it appears the negative of this hypothesis is actually the scientific one, i.e. cables do not sound different. So it looks like you have to catch one (i.e. show that cables can sound different) to disprove this Negative Snorg Hypothesis that I for one hold onto as the front-running candidate for the Truth.

I was thinking more of the fact that it was long known that the planet Uranus did not quite follow the orbit predicted by the law of gravity and the information at hand. Did the scientists conclude that the law of gravity was invalid? Did this falsify the law of gravity? No, it worked quite well in so many areas that the scientists were loath to come to such a conclusion. They decided that there must be some factor affecting the orbit of Uranus of which they were not aware. So, they started looking for another planet and eventually found Pluto. This was an ad hoc hypothesis, and as it turns out, it was a correct one.

One trouble with Snorgs are that they are undefined, and so how could one know whether one had actually found one.

[Monstrous Mike]

One trouble with Snorgs are that they are undefined, and so how could one know whether one had actually found one.

You wouldn't. You would first have to find one and then go from there to see what you have. But I still hold this analogy is true with cable sonic differences. You would first have to find a difference, and then explore the nature and cause of the difference. But right now, Snorgs and cable differences are both claimed to exist but we really don't have proof which makes it impossible to speculate on further information about either.

[okiemax]

You wouldn't. You would first have to find one and then go from there to see what you have. But I still hold this analogy is true with cable sonic differences. You would first have to find a difference, and then explore the nature and cause of the difference. But right now, Snorgs and cable differences are both claimed to exist but we really don't have proof which makes it impossible to speculate on further information about either.

It looks to me like the primary purpose of the Snorg analogy is to disparage people who claim to hear a difference in cables rather than show how you can't prove a negative. You could substitute "God" for "Snorg," and the analogy would be as accurate, but that wouldn't belittle cable yeasayers, and your purpose wouldn't be served. Why beat around the bush? If you believe anyone claiming to hear a difference in cables is imagining the difference, why not just say so.

Why does the absence of positive results "make it impossible to speculate on further information? I can understand questioning whether such an effort holds promise, but I don't see it as being "impossible." And wouldn't the possiblity of insufficient testing and/or invalid testing methodology make further investigation worthwhile?

[skeptic]

Scientific method can be explained in many ways but regardless of how you do it, in the end, it boils down to this, you are going to make some tentative conclusions about the physical universe as the result of a theory and a test to see if you can prove the theory wrong through logical conclusions from observations of the test. It isn't possible to prove that a notion is right, only that within the limited scope of the test (experiment), it is not wrong and that is an entirely different thing. That is why scientific theories no matter how well accepted are ALWAYS open to challenge. When a scientist accepts something as absolute, he is no longer a scientist. Sometimes this happens and it can be caused by ego, profit motive, prejudice, religious conviction, but for whatver reason it happens, it is not science.

The concept of randomness is a non scientific human invention. If you believe in cause and effect, in other words in a rational universe, then no event is random. Randomness expresses the human need to generalize especially a large number of events or a large number of samples, or something that occurs over a very long time. For example, we say that atoms of gas are randomly dispersed in a container but each atom or molecule was the subject of a specific history of collisions which set them on their exact course and it is in a specific location with a specific velocity at any given time because it cannot be anywhere else. Were a meteorite to hit earth over the next million years, the size, speed, and time of the collision will not be random, merely the result of variables beyond current human capacity to predict accurately.

The atoms which make up our body, our brains, our surroundings behave in the same way. The notion of free will is a complete illusion. No matter how strongly you believe you have choices in what you do, what you say, what you think, you are wrong. That is strictly a religious notion. The only other alternative explanation of existance to science or religion is existentialism.

As for the scientific testing of audio reproduction equipment, it looks to demonstrate that two pieces of audio gear which perform the same general function don't sound alike and to understand why by integrating the observed differences with accepted theories of electrical engineering and acoustics. That is about all that can be done. That these electrical and acoustical theories are incomplete or contain inaccuracies at the current state of knowledge is highly likely but it is the best we can do. The alternative is to take claims on blind faith, a method that for most people has been relatively unreliable.

[E-Stat]

That these electrical and acoustical theories are incomplete or contain inaccuracies at the current state of knowledge is highly likely but it is the best we can do. The alternative is to take claims on blind faith, a method that for most people has been relatively unreliable.

True, yet there are a number of engineering venues where such "blind faith" claims by experienced individuals is widely accepted. One of my other passions is riding motorcycles. On a four hour coast to coast flight, I read the latest Motocyclist magazine that tested four new uber performance liter sport bikes. Here are 400 pound bikes that literally outrun an FA-18 Hornet in a quarter mile. (The editor was given a ride in one by "Boss", the commander of the Blue Angels). Here are guys who have extensive on hands experience riding these rockets at triple digit speeds. There are none faster. They performed quite a few performance metrics and found that lap times on a racing course varied only from 1:28:24 to 1:29:40. Barely over one second. Quarter mile acceleration figures varied less than a fifth of one second. Despite this utter measureable parity, the four bikes exhibited very individual characters as to how they got there. Engine flexibility, steering turn-in, cornering stability, weight transition, ability to trail brake, etc., etc. all were distinctly different. And of the four testers who hammered these 175 mph bullets over a period of two weeks, the conclusions as to which was best were likewise varied.

I submit the same situation exists with audio. There are audible differences. Period. Audo journalists who likewise have extensive exposure to the best gear surprisingly come to similar conclusions as to the sonic differences among various components. As you observed, however, the first question is "which is more accurate?" The answer seems to vary from person to person because in a world where no component is perfect, one judges by their priorities. I'm not sure there will ever come a time that there will be "conclusive numbers" that fully evaluate any kind of high performance machine. It surely hasn't occurred in over a hundred years of automotive and motorcycle development.

rw

[skeptic]

There are rational and irrational ways to make choices. An irrational way is to buy something because you just like it, no other particular reason. Or the reason may have nothing to do with one thing being better at what it is supposed to do as its primary function. Or somewhat irrationally, you might trust the guy who told you it was very good and something you would want to own, especially if you liked his advice before. I often buy wine that way, not having the opportunity to taste it in advance so I listen to what some respected reviewer might say. Or maybe you like the way it looks or smells or feels in your hands.

But to people who buy wire for the sole function of connecting high fidelity sound equipment together and who may try to hide the wires because they don't particularly like their looks, how well it performs its function is the ONLY criteria by which it is judged. Now you would think that if someone wanted to sell you something that is more expensive and which they claimed did its job better, they would have some evidence to prove it. But the market isn't entirely rational and so these people who make and sell that stuff have done quite well for themselves. And of course there is a bandwagon effect because when lots of people like it, even reviewers and engineers can be intimidated by a minority of loud voices. Sometimes I've heard otherwise perfectly rational electrical engineers say that they thought that the Monster Cable they bought made a slight improvement in their sound systems. By their sheepish manner they don't want to go against the crowd, or admit that they might have made a mistake, but they don't want to endorse it either.

It either does or doesn't make an audible difference for at least some people. This can or can't be proven in double blind tests. It either can or can't be demonstrated on a lab bench. Limitations of electrical engineering knowledge notwithstanding, what we do know tells us that the answer is that it can't. And for the umpteenth time, as far as I am concerned, it's up to the guys making and selling it to back up THEIR claims of superiority with some credible evidence, not testimonials. After 25 years in the market, they haven't. Why? Two reasons. One is that they probably can't. The other is that they definitely don't have to. Suckers will believe whatever they want and don't confuse them with whatever facts we do have. A lot of people held out forever believing the tobacco companies who told them that smoking cigarettes doesn't cause lung cancer even as they were dying of it. And for all I know, there may still be a "Flat Earth Society" just as there was 50 years ago before men traveled in space. ( http://www.alaska.net/~clund/e_djubl...rthsociety.htm ) It's your money, waste it any way you like.

[mtrycraft]

.

I submit the same situation exists with audio. There are audible differences. Period. Audo journalists who likewise have extensive exposure to the best gear surprisingly come to similar conclusions as to the sonic differences among various components. As you observed, however, the first question is "which is more accurate?" The answer seems to vary from person to person because in a world where no component is perfect, one judges by their priorities. I'm not sure there will ever come a time that there will be "conclusive numbers" that fully evaluate any kind of high performance machine. It surely hasn't occurred in over a hundred years of automotive and motorcycle development.

rw

You may submit anything you like. But is it relevant? Does it have meaning? No, your cycle and car comparison is not the same as audio. Too bad you don't know this yet.
And I suppose your highly regarded journalists are not biased? That they can place a hold on their biases? LOL
How well do they do under bias controlled conditions? after all, if they trust their hearing so much, it should be no challenge for them to perform just as well.
But, reality is different from fantasy land. LOL. Enjoy.

[E-Stat]

And I suppose your highly regarded journalists are not biased? That they can place a hold on their biases? LOL

Of course they are biased. I am biased. So are you. All of us are regarding everything we choose that is not perfect. I like Dell computers not only because of their reliability but also because of the keyboard feel. As a touch typist, I find other keyboards quite clunky. That aspect may have little relevance to you. The point is that when someone thoroughly articulates why it is that they prefer brand x to brand y, now they are communicating useful information to a potential buyer.

rw

[Rockwell]

Of course they are biased. I am biased. So are you. All of us are regarding everything we choose that is not perfect. I like Dell computers not only because of their reliability but also because of the keyboard feel. As a touch typist, I find other keyboards quite clunky. That aspect may have little relevance to you. The point is that when someone thoroughly articulates why it is that they prefer brand x to brand y, now they are communicating useful information to a potential buyer.

rw

It's only valuable if brand x and y are actually different from one another for the properties being evaluating. For wires, audible differences haven't been established, and user A's evaluation of the audible properties of a wire are to totally useless to user B because A's testing is flawed and the testing conditions of A cannot be duplicated by B. So, B listening to A yak about the sound of wires is useless and misleading to B.