Thursday, November 15, 2007

The Perils of Anecdotal Evidence

Most of this post is from the wiki post on anecdotal evidence :

"Anecdotal evidence is an informal account of evidence in the form of an anecdote or hearsay. The term is often used in contrast to scientific evidence, which are types of formal accounts. Anecdotal evidence is often unscientific because it cannot be investigated using the scientific method. Misuse of anecdotal evidence is a logical fallacy and is sometimes informally referred to as the "person who" fallacy ("I know a person who..."; "I know of a case where..." etc. Compare with hasty generalization). Anecdotal evidence is not necessarily typical; statistical evidence can more accurately determine how typical something is. Psychologists have found that people are more likely to remember notable examples than typical examples[1].

In all forms of anecdotal evidence, testing its reliability by objective independent assessment may be in doubt. This is a consequence of the informal way the information is gathered, documented, presented, or any combination of the three. The term is often used to describe evidence for which there is an absence of documentation. This leaves verification dependent on the credibility of the party presenting the evidence.

In science, anecdotal evidence has been defined as:

* "information that is not based on facts or careful study" [2]
* "non-scientific observations or studies, which do not provide proof but may assist research efforts" [3]
* "reports or observations of usually unscientific observers" [4]
* "casual observations or indications rather than rigorous or scientific analysis" [5]
* "information passed along by word-of-mouth but not documented scientifically"

Researchers may use anecdotal evidence for suggesting new hypotheses, but never as supporting evidence.

Anecdotal evidence is often unscientific or pseudoscientific because various forms of cognitive bias may affect the collection or presentation of evidence. For instance, someone who claims to have had an encounter with a supernatural being or alien may present a very vivid story, but this is not falsifiable. This phenomenon can also happen to large groups of people through subjective validation.

Anecdotal evidence is also frequently misinterpreted via the availability heuristic, which leads to an overestimation of prevalence. Where a cause can be easily linked to an effect, people overestimate the likelihood of the cause having that effect (availability). In particular, vivid, emotionally-charged anecdotes seem more plausible, and are given greater weight. A related issue is that it is usually impossible to assess for every piece of anecdotal evidence, the rate of people not reporting that anecdotal evidence in the population.

A common way anecdotal evidence becomes unscientific is through fallacious reasoning such as the post hocfallacy, the human tendency to assume that if one event happens after another, then the first must be the cause of the second. Another fallacy involves inductive reasoning. For instance, if an anecdote illustrates a desired conclusion rather than a logical conclusion, it is considered a faulty or hasty generalization. [9] For example, here is anecdotal evidence presented as proof of a desired conclusion:

"There's abundant proof that God exists and is still performing miracles today. Just last week I read about a girl who was dying of cancer. Her whole family went to church and prayed for her, and she was cured."

Anecdotes like this are very powerful persuaders, but they don't prove anything in a scientific or logical sense. [10] The child may have become better anyway and this could be an example also of the regressive fallacy. Anecdotal evidence cannot be distinguished from placebo effects. [11] Only double-blind randomized placebo-controlled clinical trials can confirm a hypothesis.

Sites devoted to rhetoric [12] often give explanations along these lines:

Anecdotal evidence, for example, is by definition less statistically reliable than other sorts of evidence, and explanations do not carry the weight of authority. But both anecdotal evidence and explanations may affect our understanding of a premise, and therefore influence our judgment. The relative strength of an explanation or an anecdote is usually a function of its clarity and applicability to the premise it is supporting. [1]

By contrast, in science and logic, the "relative strength of an explanation" is based upon its ability to be tested, proven to be due to the stated cause, and verified under neutral conditions in a manner that other researchers will agree has been performed competently, and can check for themselves."

There are so many problems with anecdotal evidence for the existence of god. It is easiest to show those problems with specific examples. So, provide some if you have them.

One of the biggest problems is non-falsifiability. In other words, God is covered no matter what the outcome is. If one asks god for something, and you get it, he gets the credit. If you don't get it, either it was not god's will or god does not respond to our requests like a circus act or we were unworthy, etc. If we don't ask for anything, and we get something good, well, god sometimes blesses us even when we don't believe in him or it was going to happen anyway. If we don't ask and we don't get it, well, that is just what happens or god is punishing you. There is no control condition. There is no condition in which if such and such outcome happens, we have shown the premise of god's existence is false. This makes a true experiment impossible, and without that you only have correlational data, and correlations cannot prove causation, for there are many factors that can explain correlations.

Furthermore, the supposed correlations may be only perceived correlations and not true correlations due to selective memory, confirmation bias, and the availability heuristic. We fall victim to these effects without even knowing it and even when we are trying very hard not to. Sincerity does not mean accurate reporting.

Conditions of the experiement must be written down before the experiement is run and one must be specific about what constitutes failure or success. An independent party needs to write down the outcome. Skeptics need to review the evidence to see if the outcome actually satisfies the conditions set out in the beginning. A close call counts as a miss.

You see people often say that they watch a video and it was just like the vision they had earlier that week. Was the vision written down before the video was watched? If not, how do we know that your mind is not merging your memories or giving you a false de javu? We don't. You are sure it is not, but that doesn't count for squat. If I had you here with me I could implant false memories very easily. Our minds are not tape recorders. Each time we retrieve a memory different pieces of it are reassembled anew. It is very easy to be certain and wrong at the same time.

And we tend to remember our hits and forget our misses. We forget about all the premonitions we have that never came true.

We also forget how we subconsciously pick up on certain clues and warnings from our environment.

For every miracle, there are many possible explanations. We will never know what the true explanation of a past event was, but we can design experiments that replicate the situation and rule out many of the possible explanations. So, since we can never know the true explanation for past events, we can never be justified in being certain that god is the explanation.

We sometimes marvel at the odds that somethings could happen without god's intervention. What we forget is that we may not have to explain as much as it first appears. The event may not have happened quite as the person is telling you. The probability of what really happened may not be that rare. And we mustn't forget the law of very large numbers, in that even very improbable events will happen if given enough trials. And we need to remember that we have a tendency to way underestimate the likelihood of some events occuring. And our inability to imagine alternative explanations says more about our ignorance than the true possibilities.

No comments: