In their evaluation report, they stated that the Within Precinct Error was "most likely due to Kerry voters participating in the exit polls at a higher rate than Bush voters" (page 3). They further state, "It is difficult to pinpoint precisely the reasons that, in general, Kerry voters were more likely to participate in the exit polls than Bush voters. There were certainly motivational factors that are impossible to quantify, but which led to Kerry voters being less likely than Bush voters to refuse to take the survey. In addition there are interactions between respondents and interviewers that can contribute to differential non-response rates" (page 4).
The evaluation report lists several factors that were associated with differences in WPE, including interviewer distance from the polls; interviewer age and education; number of precincts at the polling place; and sampling rate at the polling place. It does not offer a master theory of the discrepancy.
[1/6/07: TIA asks, by my count, three questions: (1) "What about the fact that discrepancies are caused by uncounted and miscounted votes? Why don’t the exit pollsters even mention that?" Actually, as noted in 4.3, the evaluation report addresses the possibility of miscounted votes as early as page 3. (2) "Why did they ignore the fact that their own data indicated a slight PRO-BUSH BIAS?" Umm, because their own data don't indicate a slight pro-Bush bias. TIA may be referring to the argument discussed in 4.4 below. (3) "Why didn’t they provide a rationale for the fact that paper ballot precincts had a low average WPE compared to the touch screens, levers and optical scanners?" They did, as TIA must know. I have added a further discussion to the FAQ as point M.5.]
First of all, it is a name coined by critics. What the pollsters concluded in the evaluation report was simply that Kerry voters apparently participated at a higher rate. That statement doesn't entail that Bush voters recoiled from the interviewers. It is possible that some Bush voters did steer around interviewers, and/or that some Kerry voters steered toward interviewers, and/or that some interviewers subtly favored Kerry voters over Bush voters. The statements I quoted above do emphasize non-response bias -- i.e., that Kerry voters were less likely to refuse than Bush voters were. Sampling bias by interviewers (approaching more Kerry voters than Bush voters) could also contribute to a difference in participation rates.
The evaluation report never refers to "reluctant Bush responders." In fact, it doesn't venture any specific motivations -- reluctance, eagerness, fear, enthusiasm, defiance, shyness, or anything else. This reserve is appropriate because, after all, the exit pollsters did not interview the non-respondents, and therefore they cannot know specifically why the non-respondents did not respond. (Indeed, even being able to interview the non-respondents might not help very much.)
[TIA claims that "the exit pollsters had conjure up an explanation of why the polls were 'wrong' without considering the very real probability of fraud." This is untrue, and arguably dishonest, since the exit pollsters address fraud as early as page 3 of their evaluation report. I suppose what TIA really means is that if the exit pollsters saw things as clearly as he does, they would have to agree with him.]
I will present several lines of argument that participation bias accounts for much of the exit poll discrepancy, and that fraud does not. (By the way, concluding that fraud doesn't account for the exit poll results isn't the same as ruling out fraud generally.) The pollsters had several good reasons to suspect from the outset that participation bias was important. First of all, non-response bias -- and other forms of non-sampling error -- are ubiquitous concerns in survey research. Survey researchers do not assume that their data are accurate within the computed "margin of error," because they are well aware of everything that can go wrong. Second, the national exit polls' history of overstating the Democratic vote share in precincts, as described in point 3.2, gives additional reason to consider participation bias likely.
Also, we know that the exit pollsters tested a specific hypothesis about massive fraud: that millions of votes were stolen on electronic voting equipment (Direct Recording Electronic machines and/or optical scanners). In the evaluation report, the pollsters state, "Exit polls do not support the allegations of fraud due to rigging of voting equipment. Our analysis of the difference between the vote count and the exit poll at each polling location in our sample has found no systematic differences for precincts using touch screen and optical scan voting equipment. We say this because these differences are similar to the differences for punch card voting equipment, and less than the difference for mechanical voting equipment." (pages 3-4) That is, the largest exit poll discrepancies on average were in precincts with lever machines. Granted, it might be possible to steal many, many votes on lever machines, but as far as I know, no one predicted or suspected it.
Again, below I present other lines of argument that tend to point toward participation bias, and away from fraud, as explaining the exit poll discrepancies.
[1/6/07: TIA: "There was no difference in WPE between touch-screens and optical scanners? They are really reaching with that one. That only means that both machines were rigged." How does he know it means that? Presumably because he 'knows' that the exit polls are accurate. How does he 'know' that? (Hmm.) Why does he leave out the references to punch cards and mechanical voting equipment? Presumably because he has no particular theory about how those might have been "rigged" and prefers not to think about it. Ah, but what about hand-counted paper ballots? I will discuss those in the miscellaneous section.]
No, and I'm amazed how much mental effort has gone into elaborating this very weak argument. Here is the argument: The reported completion rate in precincts that voted 80%+ for Bush ("Bush strongholds") was 56%. The reported completion rate in precincts that voted 80%+ for Kerry was only 53%. So how can anyone claim that Kerry voters had a higher completion rate than Bush voters? Doesn't it appear that Bush voters had a higher completion rate?
One can work a lot of math into the argument, but basically the math is irrelevant. It seems that either one finds this argument compelling or one doesn't, regardless of the numbers attached. As a matter of logic, the argument has no force. (The logical error is called the "ecological fallacy.")
It is perfectly possible -- even likely -- that both Kerry voters and Bush voters completed the exit polls at a somewhat higher rate in "Bush strongholds" than their counterparts in "Kerry strongholds." The evaluation report suggests (on page 31) that "hypothetical completion rates of 56% among Kerry voters and 50% among Bush voters overall would account for the entire Within Precinct Error that we observed in 2004," but no one believes that those rates could apply everywhere. Indeed, the reported state-level completion rates ranged from 42.7% in South Dakota to 66.7% in Tennessee (pages 54-55) -- so we can be pretty sure that that both Kerry and Bush voters in Tennessee participated at a higher rate than their counterparts in South Dakota! Since Kerry did better in Tennessee than in South Dakota, can we conclude that Kerry voters had a higher completion rate overall? Absolutely not -- and if even the suggestion seems strange to you, then the "stronghold" argument should seem strange for the same reason. Comparing the overall completion rates in "Bush strongholds" and "Kerry strongholds" simply cannot tell us whether Kerry or Bush voters participated at a higher rate within either -- much less nationwide.
[1/6/07: TIA retorts: "Once again, you confound the basics." I guess that is TIA-speak for "I don't have to answer that!" -- and, indeed, he doesn't. TIA links to a chart that is almost uninterpretable (he mysteriously calls it a "linear regression graph," from which one might expect a scatterplot, but no such luck). I believe that the graph is intended to establish that states where Kerry did best on average had lower average completion rates that states where Bush did best. Again, this is true but irrelevant. To repeat myself, more or less: it is perfectly possible that both Kerry voters and Bush voters completed the exit polls at a somewhat higher rate (on average) in "red states" than their counterparts in "blue states." TIA wants you to stop thinking about this and to look at his chart instead. Go ahead, look, but don't stop thinking.]
First to explain the "problem": The tabulation of the national exit poll at 7:33 pm on election night, based on 11,027 respondents, indicated that Kerry had a 3-point edge. The tabulation was later updated; a version time-stamped 12:22 am (just after midnight) showed 13,047 respondents, and one can infer that Kerry still leads by 3 points. The final tabulation, still available on CNN.com, reports 13,660 respondents and shows Bush ahead by approximately 3 points, as in the official returns.
At times TruthIsAll and others have suggested that these results are "impossible" in the sense that an additional 613 respondents cannot account for the shift from Kerry to Bush. Indeed they cannot. As I explained above, the tabulations are periodically updated in line with the projections -- and, therefore, in line with the vote counts.* The tabulation would have been updated even if there had been no additional respondents. So the whole idea of "impossible changes" is a red herring.
*[1/6/07: TIA complains that I "fail[] to mention that the FINAL EXIT POLL WAS FORCED TO MATCH THE VOTE COUNT - COME HELL OR HIGH WATER." Actually, I had written that the tabulations were "periodically updated... in line with the official returns," so I don't know what TIA thinks I failed to mention. See also 4.6, immediately below. (I've changed "official returns" to "vote counts" because the numbers available on election night are not "official" returns, although I'm not sure that "vote counts" is clearer on its face.) TIA goes on to claim that "'HOW VOTED IN 2000' WEIGHTS... WERE USED TO MATCH THE VOTE," which -- as he has been told repeatedly -- is simply untrue, and another rookie mistake. Or, perhaps, no longer a "mistake" once he refuses to stop making it. For further discussion of the "How Voted in 2000" issue, see part 5.)
If the vote counts are more accurate than the exit polls -- and bear in mind that exit polls have been (presumably) wrong in the past -- then weighting to the vote counts should, generally, provide more accurate tabulations. Weighting survey results to match known characteristics of the population (in this case, the voters) is called "post-stratification," and it is a standard practice. Of course, if the exit polls were more accurate than the vote counts, then weighting to the vote counts would provide less accurate tabulations. But even if one believes that they were, I see no reason to conclude that the exit pollsters thought so.
[1/6/07: TIA asks, "...why were MATHEMATICALLY IMPOSSIBLE 43%Bush / 37% Gore weights used in the Final Exit Poll?" As I noted under 4.5, they weren't. It is difficult to believe that TIA is actually still fundamentally confused about the meaning of "weights." I have no idea why TIA insists on getting basic facts wrong. As for the substance of the argument, again, refer to part 5.]
Not that I can see. The original tabulations were online for many hours, at the hardly obscure CNN.com. Exit pollsters commented on the results for reporters, and Warren Mitofsky appeared on PBS's NewsHour three days after the election. The exit poll discrepancies were not secret.
[1/6/07: TIA says, "The preliminary exit polls notes say that it is 'not for public viewing'. Without the 12:22am download from Jonathan Simon, we would never know the true exit poll results." I repeat: the tabulations were publicly posted at CNN.com. TIA cannot refute this fact, so he points to some other document that was "not for public viewing." This document has nothing to do with Jonathan Simon's downloads of the tabulations that (did I mention?) were publicly posted at CNN.com. (Incidentally, the tabulations are not the "true exit poll results," since they incorporate expectations based on pre-election polls.)]
There are several. One of my favorites is based on TruthIsAll's observation: "Based on the pre-election polls: 41 out of 51 states (incl DC) deviated to Bush. Based on the exit polls: 43 out of 51 deviated to Bush." As I discussed in Point 1, TruthIsAll's analysis of the pre-election polls is idiosyncratic: for instance, he estimated an 85% probability that Kerry would win Ohio, although only one poll out of the last ten put Kerry ahead there. Nevertheless, if the two sets of deviations tended to coincide -- if, for instance, Bush tended to do much better than the pre-election polls in the same states where he did much better than the exit polls -- then one might judge that both sets of results tended to point to fraud. (Note: this argument assumes that the extent of fraud varied from state to state, and that the differences in exit poll discrepancies partly reflect differences in fraud.)
Interestingly, as I show here, the relationship between these two deviations (official vs. pre-election polls, and official vs. exit polls) is flat to negative. That is, if anything, exit poll "red shift" overstating Kerry's support was largest in states where he did better than predicted in pre-election polls, or at least no worse. These results are calculated using TruthIsAll's own pre-election poll measures! While many people (including TruthIsAll) have asserted that pre-election polls may be biased, no one has explained why some state polls would be more biased than others, in a pattern that would cancel out the evidence of fraud. The lack of a positive correlation between these two deviations isn't strong evidence against fraud in any particular state, but it appreciably weakens the case that the exit polls point to fraud around the country. (A more detailed exploration of the data is here.)
Another strong reason to think that the exit poll discrepancies do not point to fraud involves the "swing" statistic. "Swing" is the percentage change in Bush's performance from 2000 to 2004, in each exit poll precinct. Swing tends to be fairly small, generally within a range of plus-or-minus 10 points. Generally, if an exit poll "red shift" of (say) 30 points tended to indicate 30-point fraud favoring Bush, then one would expect swing favoring Bush to be larger in such precincts than in precincts where the exit polls closely matched the official returns or even overstated Bush's support. But in fact, as I show here, there is essentially no correlation whatsoever between "swing" and "red shift." This result is hard to reconcile with fraud in the exit poll precincts, unless the fraud was precisely targeted in precincts where Bush would otherwise have done worse than in 2000.
[1/6/07: TIA says: "The 'swing' argument is getting old." In other words, as in 4.4, "I don't have to answer that!" Oh-kay. TIA really doesn't want you to think about these arguments. I bet he really, really doesn't want you to look at a scatterplot. Here's one of them (here, again, is the link for a better look and discussion):
The states with the biggest red shift are toward the bottom of the plot; the states where Bush did best compared to pre-election polls are at the left. If red shift measures fraud or miscount, why does Bush not tend to do better (compared to pre-election polls) in states with large red shifts than in states with blue shifts? (The best answer I can imagine is that the actual miscount was essentially proportionally equal in every state. Interesting if true, but I would like to hear it from TIA himself.) Here's another (again, see the full discussion here):
If red shift measures fraud, why does Bush not tend to do better (compared to predicted values based on his 2000 performance) in precincts with large red shifts? TIA's response: the "argument is getting old." Ah.]
Yes, beyond the facts that participation bias is common, that past exit polls have overstated Democratic performance, and that the exit poll discrepancies don't correlate with pre-election poll discrepancies, "swing" from 2000, or electronic voting machine use, there is also some evidence indicating participation bias in 2004. Specifically, some patterns in the exit poll discrepancies fit well with participation bias.
For instance, "red shift" is much greater in precincts where interviewers stood further from the polling place. The average WPE increases from -5.3 where interviewers stood inside the polling place to -12.3 where interviewers stood more than 100 feet away (evaluation report, p. 37). This result is consistent with the conjecture that Kerry voters were more eager to participate in the exit polls than Bush voters were -- and this disparity was more manifest in polling places where it was easier to avoid the interviewers entirely. The recorded completion rate falls from 59% where interviewers stood inside the polling place to 43% where they were more than 100 feet away; both refusal and "miss" rates go up. At that distance, the "random sample" becomes largely voluntary.
(Some people have objected that interviewer distance cannot account for the error in the poll because the WPE is so large even when interviewers stood inside the polling place. This misses the point. Edison/Mitofsky never asserted that interviewer distance accounts for non-response bias. If Bush voters and Kerry voters are equally willing to participate in the survey, then distance should not matter. But if they differ overall in their willingness to be interviewed, then it makes sense that greater distance would accentuate the difference.)
"Red shift" is also much larger at polling places with high interviewing rates -- that is, where interviewers were supposed to skip more voters between interviews (evaluation report, p. 36). Interestingly, however, the reported completion rates were not lower at these polling places. So it appears that interviewers at these polling places sometimes (probably unawares) ended up interviewing Kerry voters when they "should have" interviewed Bush voters. Indeed, there is anecdotal evidence that some interviewers did not rigorously enforce these interviewing rates, but rather sometimes interviewed voters who seemed more cooperative.
Red shift also varies with certain interviewer characteristics, such as age and education (evaluation report, pp. 43-45); it appears that interviewers who were college students were especially prone to biased results. Voters may have tended to assume that college students would be Kerry voters, and gravitate toward or away from them accordingly.
[1/6/07: TIA repeats that completion rates were lower on average in states with higher Kerry proportions, again ignoring the explanation in 4.4 of why this fact proves nothing. He also completely ignores every point I've made here.]
You could call it that, or you could call it scientific reasoning on the basis of incomplete evidence. William Jennings Bryan once said that "Darwinism... is only a guess and was never anything more" -- but Bryan made no systematic attempt to consider how well "Darwinism" explained a broad variety of evidence, compared with the explanatory power of alternative theories. Attributing outcomes to the unfathomable will of God "explains" them by eschewing any attempt to understand them. In practice, attributing exit poll outcomes to fraud often suffers the same limitation. People who insist that the exit polls evince fraud have made little or no attempt to explain, for instance, why the exit poll discrepancies vary with interviewer age and education, or why the exit polls point to double-digit fraud in New York, a lever-machine state where Bush had no chance of winning, and where the official result is very close to pre-election predictions. Given all the respects in which participation bias fits the data, and in which massive fraud does not, it is hard to understand how anyone can argue straightfaced that participation bias is the more speculative theory.
[1/6/07: TIA objects to my comments about New York, but seems uncertain why. First he says, "The exit polls point to double-digit fraud in NY? That's misleading." Then he offers up some numbers that suggest, to him, that "Kerry won by 64-35% - within 1% of the Exit Poll." OK, so Kerry won by 29, and the official count says he won by 18.3... what part of "double-digit fraud" is misleading? He concludes: "And yes, there is evidence that Lever machines are vulnerable to miscounts...." But, TIA, I don't want to know whether it is hypothetically possible that New York lever machines conceivably could have been hacked. I want to know whether over 700,000 net votes actually were stolen here, how many people would have had to conspire to do it, and why they bothered. And, if Kerry won New York by 29 points, I also want to know why the four pre-election polls in the last week were also off by double digits, every single one of them. Why, TIA, are you so profoundly incurious about these questions?]
No. As noted earlier, many forms of fraud may be compatible with the exit poll results. However, it seems hard to reconcile massive, widespread fraud -- on the order of many millions of miscounted votes -- with the exit poll results unless one begins by discounting the details of the exit poll results. By and large, the exit polls point to fraud in bizarre places (Delaware?) and sizes (how likely is it that Kerry would have won New Hampshire by double digits -- or New York by over 30 points?). One can explain away some of these results individually, but the entire pattern simply isn't very consistent with massive, widespread fraud.
No, depending on what one means by "steal." In particular, I think it is at least possible that some combination of vote suppression (purges, long lines, intimidation, etc.) and uncounted votes cost John Kerry a victory in Ohio, and therefore in the election. (Obviously "uncounted votes" can be regarded as a form of vote suppression.) I doubt it, but I am not arguing against it here. Many forms of vote suppression would not even affect the exit poll results, if they discouraged people from coming to the polls in the first place. It is easier to believe that Kerry "could have" won Ohio than to believe that he should have won by over 6 points despite vote suppression, as the exit poll indicates.
Please note: I think that efforts to prevent people from voting, or to prevent their votes from being counted correctly, are despicable regardless of whether they affect the outcome. [I added the word "correctly" to clarify my meaning.]
Previous: (3) Describing the Exit Poll Discrepancies
Next: (5) Comparing 2004 to 2000
Up to Table of Contents