What is "evidence"

When should an observation be regarded as evidence for a proposition?

(...)

"Evidence" for a proposition is any thing that increases the estimate of the probability of the truthfulness of the proposition. Note that this definition means that what is evidence is dependent on the agent that estimates the probability. That is why the question in the first sentence is not the simpler question "When is an observation an evidence for a proposition?" The answer to this question is actually agent-dependent, unless it is interpreted as if it is the question in the first sentence.

(...)

An important point to note is that the second part is important as well: an observation that is compatible with other plausible states of affairs should not be taken as an evidence for a proposition.

(...)

Even though the logic is obvious, there is a strong tendency to ignore the second condition above. It is quite common for people to regard as evidence for some proposition an observation which is compatible with many other plausible states of affairs. In some cases, they justify it by considering only restricted set of the other plausible states of affairs (I call this the "Misanalyzing the 'Null Hypothesis'" error in Reasoning errors). In other cases, they don't bother to justify it at all, and seem to do it simply because they like the proposition, and hence accept automatically the observation as evidence (implicitly doing the "conclusion-validation" error, Reasoning errors).