Bias and swing in the 2004 NEP exit polls: do the exit polls indicate fraud?
Mark Lindeman, lindeman@bard.edu
fourth edition 7/28/08 (minor edits, date references removed)
Warren Mitofsky has given me permission to post two new scatterplots that he presented in Philadelphia on Friday, October 14 as part of a 'debate' with Steve Freeman at a meeting of the Philadelphia chapter of the American Statistical Association. (I put 'debate' in scare quotes because the format was Dueling PowerPoints, with limited opportunity to respond to each other's points.) One can find Freeman's argument that the exit polls contain "neglected correlation after neglected correlation pointing toward fraud," with supporting slides, here. Freeman argues that the exit polls "indicate" that John Kerry won the 2004 election by perhaps 6.5 million votes; Mitofsky finds no evidence for fraud in the exit polls. I will present my views about this debate more extensively as time permits (short version: I basically agree with Mitofsky), but for now I limit myself to introducing these scatterplots, with minimal background. I take sole responsibility for the descriptions of the scatterplots.
Most respondents in the 2004 NEP exit polls reported having voted for John Kerry, but Bush won by 3 million votes in the official count. This discrepancy is called a "red shift", i.e., Bush does better in the vote compared to the exit polls. Red shift can also be characterized as bias favoring Kerry, i.e., Kerry doing better in the exit polls compared to the vote. In some precincts and states there was "red shift," in others "blue shift" (i.e., Kerry did better in the vote than he did in the exit polls), but the red shift far outweighs the blue shift overall. Either the polls were wrong (beyond the scope of random error), the vote count was wrong, or both.
So, possibly much of the "red shift" -- Bush doing better in the vote count than in the exit polls -- represents vote fraud favoring Bush. If that were true, I for one would expect Bush to do "better" in places with red shift than in places with blue shift (or no shift)*. But what do I mean by "better"? Suppose that the fraud took the form of destroying a lot of Democratic votes in heavily Democratic areas. (Quite a few votes are discarded in every election, but to account for the exit poll discrepancy, 2004 would have had to be very unusual.) Then we would find that the red shift was greater in precincts where Bush did relatively badly -- but still better than he "should" have. Thus, we need some way or ways of estimating how well Bush "should" have done.
* This expectation assumes that vote-count fraud varies substantially from precinct to precinct. If vote-count fraud is ubiquitous and does not vary much (if at all), then exit poll shift obviously cannot effectively distinguish some precincts as more fraudulent than others; any variance contributed by fraud could be swamped by error variance. Since Freeman's arguments emphasize "neglected correlation after neglected correlation pointing toward fraud," they do not lend themselves to this conjecture that vote-count fraud is pretty much equal everywhere!
The analyses here use Bush's performance in 2000 as a basis for judging how he "should" have done in 2004. (Another approach, at the state level, would be to use pre-election polls. That yields some interesting results that I will write up soon.) Change between elections is often called swing -- where Bush did better in 2004 than in 2000, that is positive swing (from his standpoint, at least!). Do swing and red-shift bias go together?
 SLIDE 1 depicts George W. Bush's vote share in 2004 as a function of his vote share in 2000, for exit poll precincts where both figures were available. (Some details about the precincts included are given in Note 1 below.) Precincts are color-coded by their 2004 poll-vote "shift": "red shift" (Bush did better in the vote count than in the exit poll responses) is indicated by red dots, and "blue shift" by blue dots. Two best-fit regression lines are displayed: one for red-shift precincts, one for blue-shift precincts. For instance, if lots of Democratic votes were destroyed in Democratic precincts in 2004, we would expect the red-shift best-fit line to run above the blue-shift line at the lower left, indicating red-shift precincts where Bush still lost in 2004, but did much better than he did in 2000. Or, for that matter, Bush might do worse in these precincts in 2004 than in 2000, but "less worse" than in other precincts where fraud was not feasible. The question here is not whether there is swing favoring Bush, but whether Bush does better by the swing measure where there is red shift -- putatively indicating fraud -- than where there is blue shift. If not, then widespread fraud is still possible, but at best more difficult to engineer, as I argue below.
--What's that? you only see one line in Slide 1? Look closely. The red-shift line is essentially superimposed on the blue-shift line! Judging by these lines, Bush does no better (or worse) in 2004 relative to 2000 in the red-shifted precincts than in the blue-shifted precincts, either in Democratic or in Republican precincts. |
 SLIDE 2: Since some red and blue shifts are shiftier than others, the second slide depicts the actual magnitude of the exit poll error (bias) along the X axis. I explain the measure below in Note 2, but to give an idea: if Kerry's exit poll percentage were 60% but his vote percentage were only 50%, the bias would equal 0.197. As you can see, there are lots of exit poll biases larger than that in both directions. Big red-shift means large positive bias (points on the right); big blue-shift means large negative bias (points on the left). The Y axis depicts the "residual" from the first graph -- how far each precinct was above the best-fit line relating Bush's 2004 vote share to his 2000 vote share (see Note 3). Thus, precincts where Bush did better in 2004 than one would predict from his 2000 performance (based on the best fit as depicted in Slide 1) have positive residuals; where Bush did worse in 2004 than one would predict from his 2000 performance, the residuals are negative. Again, if red-shift from poll to vote count indicates fraud, one would probably also expect Bush to do better relative to 2000 in the red-shift precincts than in the blue-shift precincts. So the best-fit line would tilt upward: the red-shift "Kerry (poll) bias" precincts would tend also to have [11/2: more positive or less negative] Bush swing from 2000 to 2004.
--The best-fit line here is thin, and at first you might mistake it for a flat "zero line." Strictly speaking, the line tilts slightly downward, but essentially it is about as flat as regression lines get. Again, Bush's "swing" (here relative to the 2000-04 best fit) does not increase as red shift increases. |
Are these results compatible with massive fraud that would shift on the order of 10 million net votes from John Kerry to George W. Bush? (If this question seems unfair, see Note 4. ) Possibly. Just as "correlation doesn't prove causation," lack of correlation doesn't prove non-causation. But the failure of Bush to do better in red-shift precincts than blue-shift precincts is a conundrum for analysts who see strong evidence of fraud in the exit polls. If fraud lurks in these scatterplots, it is very well disguised.
Could well-disguised fraud lurk in these scatterplots? Critics have rightly pointed out that there is an important distinction between (a) Bush doing better than he should have because of fraud and (b) Bush having more favorable swing in fraud precincts than in non-fraud precincts. Isn't it possible that fraud could be concentrated in precincts where Bush otherwise would have done especially poorly in 2004 compared to 2000? In that case, fraud might simply bring the swing in red-shift precincts back to the same level as the swing in blue-shift precincts, as we see in the Mitofsky scatterplots, yet Bush would still have done much better in these precincts than he should have. This sounds simple, and in fact it is fairly simple, if we assume that (1) the purveyors of fraud can influence the vote in almost all precincts** and (2) they can calibrate fraud based on the actual direction of swing (perhaps by reliably predicting it in advance, or by being able to determine the actual vote counts during or after voting and alter them accordingly). (Or we might assume that the purveyors of fraud can corrupt the exit polls, or target fraud in non-exit-poll precincts -- which would seem to concede the point that the exit polls are not strong evidence of fraud.) In simulation studies, I tentatively find that fraud can be disguised to "fool" the correlational test if both these conditions obtain. But if the scope of possible fraud is limited to, say, 80% of precincts (or many fewer), or fraud cannot be focused in precincts that would otherwise have negative swing, then fraud -- on a scale that would influence the national popular vote by multiple percentage points -- becomes increasingly hard to disguise. This line of argument seems difficult to square with confident assertions that the exit polls offer strong evidence of fraud. It also seems to fit poorly, at least at first glance, with suggestions that fraud was concentrated in swing states, or Republican strongholds, or precincts with touch-screen machines, or various other hypotheses along similar lines. It remains to be seen whether these lines of analysis can be reconciled.
An Election Science Institute report has presented qualitatively similar results for Ohio in particular. While I presently believe that the exit polls paradoxically offer (inconclusive) evidence against the outright theft of the national popular vote, I do not argue that the Ohio exit poll offers strong evidence in any direction, given the small number of precincts surveyed. Arguments about fraud, vote suppression, and misconduct in Ohio are best evaluated on other grounds.
** This assumption is not a necessary condition, depending on how much fraud is being purveyed. Moderate degrees of fraud -- perhaps amounting to a one-point increase in overall Bush vote share -- could be concealed in a minority of precincts while inducing only modest expected correlations which would not be reliably detectable. (In any given state, quite substantial fraud might not be reliably detectable in the exit poll.). Nonetheless, as a general expectation, the best way to hide stolen votes from a shift-swing analysis is to steal them quite widely, at least in precincts where one otherwise would do poorly.
-----------------------
Note 1: Both slides depict results for 1250 precincts among the 1480 included in the exit poll; other precincts are excluded because vote returns were not available, because extensive absentee votes were merged with the precinct vote, because fewer than 20 interviews were conducted, or (in three cases) because the errors were extraordinarily high. Return to text
Note 2: The bias measure is an arctangent transform of relative participation ratios, as proposed by Elizabeth Liddle and myself. (I will demonstrate in a moment why we prefer a relative ratio measure to a simple percentage difference.) Specifically, bias = arctan (K/B) - arctan (1), where K = Kerry's exit poll proportion divided by his official count proportion, and B = Bush's exit poll proportion divided by his official count proportion (all excluding third-party votes). Bias is positive when Kerry does better in the exit polls than the official returns (red shift), and negative when Kerry does better in the official count (blue shift). For instance, if Kerry "won" 60% of the exit poll respondents in a precinct, but received only 50% of the votes, K = 60 / 50 = 1.2; B = 40 / 50 = 0.8; K / B = 1.2 / 0.8 = 1.5; and bias = arctan(1.5) - arctan(1) = 0.197. But if Kerry got 20% of the exit poll respondents and only 10% of the votes, K = 20 / 10 = 2.0, B = 80 / 90 = 0.89, K/B = 2.0 / 0.89 = 2.25, and bias = arctan(2.25) - arctan(1) = 0.367. (If we swapped Bush and Kerry in either of these examples, we would end up with the same level of bias, but negative instead of positive.) The bias is much larger in this second example because, in general, misestimating (or miscounting) the proportion of Kerry voters by a factor of 2 is a more startling result -- although just how startling it was would depend on the numbers of voters and poll respondents involved. Return to text
Note 3: Mitofsky hasn't provided the equation of this line, but by close inspection of slide 1, we can tell that it is close to: Bush04 = 0.97(Bush00) + 0.03. So, on average, if Bush got 50% of the two-party vote in 2000 (Bush00 = 0.5), he would be "expected" (according to the best-fit line) to get about 51.5% of the two-party vote in 2004. If Bush actually received 61.5% of the vote in 2004, the residual would be 0.615 - 0.515 = +0.1 (more or less, depending on how well I've "eyeballed" the regression line); if he received 41.5% of the vote, the residual would be 0.415 - 0.515 = -0.1. Since the regression line "predicts" that Bush will do better in all precincts, a swing of 0 (no difference between 2000 and 2004) corresponds with a slight negative residual -- in this example, 0.5 - 0.515 = -0.015. Return to text
Note 4: Inevitably, someone will accuse me of misrepresenting or trivializing the fraud debate (or the cause of election reform) by reducing it to this one argument. For now I can only say that this one argument (that Kerry may have won the 2004 popular vote by around 7 million votes instead of losing it by around 3 million) is being made; that it should be possible to address this argument without also having to address every other argument about the 2004 election; and that if the argument seems to trivialize the fraud debate, perhaps the argument should be blamed, not its critics. Return to text
