The fallacy of the test “Try it. If it works, it is true”

So many religionists believe that if they try something and it works then it denotes that something is true. Buddhists lean heavily on it. Mormons do too, as do New Agers.

Scientific research methods are designed to help scientists not be fooled by that fallacy. Let's discuss a few reasons why something might not be true, even when we try something and we get the desired outcome.

First, correlation cannot establish causation. For example, let's say these two events are correlated (i.e., they both occur in relative proximity), praying to find keys and finding the keys. Let's say that this has happened on a number of occasions. We cannot infer based solely on correlation that the praying caused us to find the keys. We might find the keys everytime whether or not we pray. Certainly there are people who don't pray (like me) who lose keys and then find them. Or, it could be that some third variable also co-occurs with the first two events, such as thinking about where you might have left the keys or you might actually look for the keys everytime you pray for them and find them. This third variable might actually be responsible for the event of finding the keys. And it could be that some other variable is responsible for causing both the praying and the finding of the keys. I can't think of one for this example, but there could be other examples in which this is the case.

Personal "tests" of a principle using do not include an adequate control group. In other words one condition in which you tried the procedure and one in which you did not. It is not enough to try each condition only once or twice as you would be taking advantage of chance. The relationship between procedure and outcome across both conditions needs to happen often enough to show that the relationship is not likely due to chance. And one needs to be on guard to make sure that you are not sabatoging the experiment by making the results what you want them to be (i.e., trying harder in the condition you want to succeed). Scientists prefer to use large sample sizes in which the subjects are unaware of the hypothesis to increase the reliability of the results.

Also, humans are very prone to remember the hits and forget the misses. For, example, they remember when their dreams have some things in common with future events, but forget all the things their dreams got wrong.

Furthermore, even if what we are doing does work, we may take the wrong lesson from it. For example, it may be that prayer actually does help us find our keys, but not due to supernatural influence but because prayer calms our minds and our calm minds are able to remember better where we left the keys. Native peoples might believe their tribal shaman has magical powers because everytime they have a headache, he gives them a special herb and their headache goes away. It might be that the herb is a natural (not supernatural) medicine or it might be that the shaman uses the persons belief to produce a placebo effect.

In summary, just because something works when you try it, it does not mean that you have established that the procedure causes the outcome, that you will get reliable results everytime you use the procedure, or that the implications that you take from the successful procedure are correct (i.e., praying works therefore there is a God, or snake oil cures broken bones, etc).

The whole idea of "this works, therefore it must be true" is how superstitions, lucky charms, and rituals like rain dances or human sacrifices are born. Some superstitions are harmless, but I am interested in discovering and understanding and teaching true causal relationships. It bugs me when people try to persuade others that something is true because it worked for them when their results may have nothing to do with what they so confidently teach.

I have no problem with people sharing with others what has worked for them in the hopes that maybe it might work for others as well. I would just caution against presupposing a causal relationship where none has been established and especially against using your fortunate results to promote some supposed truth that is in no way implied from your psuedoexperiment (i.e., that god exists because whenever you pray to him you find your car keys; or, that Mormonism is true because you often feel a good feeling when you read the Book of Mormon, etc).