From Robyn Dawes's Rational Choice in an Uncertain World:
Post-hoc fitting of evidence to hypothesis was involved in a most grievous chapter in United States history: the internment of Japanese-Americans at the beginning of the Second World War. When California governor Earl Warren testified before a congressional hearing in San Francisco on February 21, 1942, a questioner pointed out that there had been no sabotage or any other type of espionage by the Japanese-Americans up to that time. Warren responded, "I take the view that this lack [of subversive activity] is the most ominous sign in our whole situation. It convinces me more than perhaps any other factor that the sabotage we are to get, the Fifth Column activities are to get, are timed just like Pearl Harbor was timed... I believe we are just being lulled into a false sense of security."
Consider Warren's argument from a Bayesian perspective. When we see evidence, hypotheses that assigned a higher likelihood to that evidence, gain probability at the expense of hypotheses that assigned a lower likelihood to the evidence. This is a phenomenon of relative likelihoods and relative probabilities. You can assign a high likelihood to the evidence and still lose probability mass to some other hypothesis, if that other hypothesis assigns a likelihood that is even higher.
Warren seems to be arguing that, given that we see no sabotage, this confirms that a Fifth Column exists. You could argue that a Fifth Column might delay its sabotage. But the likelihood is still higher that the absence of a Fifth Column would perform an absence of sabotage.
Let E stand for the observation of sabotage, H1 for the hypothesis of a Japanese-American Fifth Column, and H2 for the hypothesis that no Fifth Column exists. Whatever the likelihood that a Fifth Column would do no sabotage, the probability P(E|H1), it cannot be as large as the likelihood that no Fifth Column does no sabotage, the probability P(E|H2). So observing a lack of sabotage increases the probability that no Fifth Column exists.
A lack of sabotage doesn't prove that no Fifth Column exists. Absence of proof is not proof of absence. In logic, A->B, "A implies B", is not equivalent to ~A->~B, "not-A implies not-B".
But in probability theory, absence of evidence is always evidence of absence. If E is a binary event and P(H|E) > P(H), "seeing E increases the probability of H"; then P(H|~E) < P(H), "failure to observe E decreases the probability of H". P(H) is a weighted mix of P(H|E) and P(H|~E), and necessarily lies between the two. If any of this sounds at all confusing, see An Intuitive Explanation of Bayesian Reasoning.
Under the vast majority of real-life circumstances, a cause may not reliably produce signs of itself, but the absence of the cause is even less likely to produce the signs. The absence of an observation may be strong evidence of absence or very weak evidence of absence, depending on how likely the cause is to produce the observation. The absence of an observation that is only weakly permitted (even if the alternative hypothesis does not allow it at all), is very weak evidence of absence (though it is evidence nonetheless). This is the fallacy of "gaps in the fossil record"—fossils form only rarely; it is futile to trumpet the absence of a weakly permitted observation when many strong positive observations have already been recorded. But if there are no positive observations at all, it is time to worry; hence the Fermi Paradox.
Your strength as a rationalist is your ability to be more confused by fiction than by reality; if you are equally good at explaining any outcome you have zero knowledge. The strength of a model is not what it can explain, but what it can't, for only prohibitions constrain anticipation. If you don't notice when your model makes the evidence unlikely, you might as well have no model, and also you might as well have no evidence; no brain and no eyes.
Part of the sequence Mysterious Answers to Mysterious Questions
Next post: "Conservation of Expected Evidence"
Previous post: "Your Strength as a Rationalist"
Can we be sure that he did not just assign a very strong prior distribution to the existence of Fifth Column? In that case, if we model Warren's decision as binary hypothesis testing with a MAP rule, say, then maybe it occurred to Warren that the raw conditional probabilities satisfied this inequality P(no sabotage | imminent Fifth Column threat) < P(no sabotage | no imminent Fifth Column threat).
But perhaps, for Warren, P(imminent Fifth Column threat) >> P( no imminent Fifth Column threat).
In this scenario, he reasoned that it was so likely that there was a Fifth Column threat that it outweighed the ease with which (absence of Fifth Column) can account for (absence of sabotage), and led him to choose the hypothesis that a Fifth Column was a better explanation for lack of sabotage.
In that case, the issue becomes the strength in the prior belief. Similar reasoning can be applied to McCarthy, or to those suggesting we're due for another terrorist attack.
I guess what I am saying is like this: maybe someone just believes we're due for another terrorist attack very strongly (perhaps for irrational reasons, but reasons that have nothing to do with a witnessed lack of terrorist attacks). Then you present them with the evidence that no terrorist activity has been witnessed, say. Instead of this updating their prior to a better posterior that assigns less belief to imminence of terrorist attacks, they actually feel capable of explaining the absence of terrorist activities in light of their strong prior.
I do agree that it would then be nonsensical to take that conclusion and treat it like a new observation. As if: Fifth Column -> they absolutely must exist and be planning something -> invent a reason why strength of belief in prior is justified -> Fifth Column's existence explains absence of sabotage -> further absence of sabotage now feeds back as ever-more-salient corroborating evidence of original prior.
Perhaps more focus should be placed on the role of the prior in all of this, rather than outright misinterpretations of evidence.