According to most observers, most pre-election polls put George W. Bush slightly ahead of John Kerry. The summary of "final trial heats" at pollingreport.com shows Bush ahead in 10, Kerry ahead in 3, and one tied. (The average margin was Bush +1.45%.) Among the "pollster vote projections" (in which the pollsters may make subjective judgments about how undecided voters are likely to vote), five favor Bush, two favor Kerry, and one is a tie. (For what it is worth, both projections favoring Kerry are attributed to Democratic pollsters; one projection favoring Bush is attributed to a Republican pollster. So among nominally neutral pollsters, Bush was ahead in four projections and tied in the fifth; the average margin was Bush +1.3%.)
Please note that every single one of these polls yielded a result within its statistical margin of error, even assuming that the polls were otherwise perfect. So they certainly don't justify high confidence that Bush was "really" ahead, much less that he was destined to win. Some folks might say that the polls revealed a "statistical tie," but that characterization throws away useful information. As I said, most of the polls put Bush slightly ahead.
Yet TruthIsAll argued, in his final Election Model, that the pre-election polls gave Kerry a 99.99% probability of winning the popular vote (as well as a 99.8% probability of winning the electoral vote). Why are his numbers so different? There are two main reasons.
First, TruthIsAll uses poll results for all registered voters, rather than for likely voters, whenever possible. Presumably he believes that the pollsters were biased toward understating Kerry support. In many surveys, people who are least likely to vote tend to favor Democratic candidates. So, for instance, the final pre-election survey by the Pew Research Center gave Bush a 3-point lead among "likely voters," but gave Kerry a 1-point lead among all "registered voters." TruthIsAll uses the figure that puts Kerry ahead, even though Pew itself headlined the report, "Slight Bush Margin in Final Days of Campaign." [More on turnout and likely voters below.]
Second, TruthIsAll assumes that undecided voters will (or did) sharply favor Kerry. (His final model assumed that 75% of undecided voters would vote for Kerry -- although, in fairness, he also considered alternative proportions.) [More on undecided voters on the next page.]
[1/6/07: TIA complains that the Pew headline "shows a pro-Bush bias since the RV poll is not mentioned." This claim is, charitably, muddled. There is no separate "RV poll": Pew's likely-voter estimate is what the analysts believe to be their best estimate. TIA goes on to assert, "In the 18 polls listed at pollingreport.com, Kerry led the 18-poll average by 47.81-47.73%." TIA transmogrifies a small Bush lead into a tiny Kerry lead primarily by suppressing the likely-voter estimates whenever registered-voter estimates are available instead. He also incorporates an online poll, sponsored by The Economist, which pollingreport.com chose not to report in its summary tables, presumably because of its nonstandard methodology.
[Perhaps anxious about this equivocal evidence, TIA refers to "final election day Zogby and Harris LV polls" which supposedly "had Kerry winning by 50-47%." The Harris poll is an online survey conducted October 29 through November 1. We might be willing to factor the online results into the averages, but we certainly can't privilege the Harris poll as a "final election day" result. The supposed election day Zogby poll is mysterious, given that a fraud-sympathetic website reproduces Zogby's projection "as of Nov. 2, 2004 5:00 pm": "Zogby International Finds: Bush at 49.4%, Kerry at 49.1%." (Zogby nonetheless projected a decisive Kerry win in the electoral college, based on two somewhat mysterious projections: Florida "Kerry .1% Trending Kerry" and Ohio "Bush 2% Trending Kerry." Why Zogby was willing to base its call on vague assertions of "trending" is a matter of conjecture.)]
Basically, those probabilities (for both state and national polls) assume that all his assumptions (for instance, about how "undecided" voters will vote) are right, and that the only source of uncertainty is random sampling error. I argue below that his assumptions are more wrong than right. They certainly aren't 100% reliable. (TruthIsAll himself suggests that the polls might be biased -- against Kerry, of course.)
[1/6/07: TIA complains that his undecided voter allocation "is based on what world-class pollsters Zogby and Harris have stated." It seems to me that even if Zogby and Harris are downright intergalactic, they have no mystical ability to descry the decisions of undecideds with 100% confidence -- and 100% confidence is what TIA's probability calculations entail. I explain in part 2 why his allocation of undecideds is not supported by historical evidence, despite his claims to the contrary.]
No, high turnout is not a reason to dismiss the likely-voter results. Most pollsters already expected high turnout. For technical reasons, it is hard to compare pollsters' various turnout assumptions to the official figures, but the survey results don't change very sharply if we change turnout assumptions by a few points. To include every respondent who claims to be registered would include many people who have very little prospect of voting.
In Pew's final pre-election poll (data available for download here), Pew scored over 62% of its respondents as "likely voters," scoring 8 or 9 on its 0-to-9 likely voter scale. (Actually, this percentage is weighted -- for instance, young respondents weigh more heavily than older ones. The unweighted results are even less favorable to Kerry.) The actual presidential turnout in 2004 is estimated by Dr. Michael McDonald at about 60.3% of the eligible voting-age population. In order for Kerry to take a slight lead in the Pew survey, one must either include all voters scoring as low as 5 on the likely voter scale -- which implies about 77% voting-age turnout -- or assume that about three quarters of all undecided voters would vote for Kerry. In order to give Kerry an appreciable lead, one has to jack up the projected turnout and allocate the vast majority of undecided voters to Kerry.
By strange coincidence, TruthIsAll did exactly that: he jacked up the turnout to include all self-reported registered voters (which, in the Pew survey, would be about 80% voting-age turnout, or around 160 million voters nationwide instead of 120 million), and allocated three quarters of undecided voters to Kerry. Presto, Kerry took the lead.
In January 2005, TIA claimed, "Any reputable pollster will tell you that in this election, RV's were a more accurate gauge of the vote." What a strange claim. Pollingreport.com reports (here and on the following "earlier" pages) likely-voter results from well over a dozen separate pollsters. Were they all disreputable? Did they all repudiate their likely-voter models?
[1/6/07: TIA objects to my assertion that he "jacked up the turnout": "That is pure fiction on your part. You are extrapolating voter turnout based on the type of poll." Wekk, no. For instance, the Pew registered-voter and likely-voter projections aren't based on two different "type[s] of poll"; they are derived from the same data, except that the likely voters are a subset of the registered voters. So it is hardly "pure fiction" to state that the registered-voter estimate implies higher turnout than the likely-voter estimate. Granted, TIA doesn't literally have to assume 77% turnout in order to rationalize using Pew's registered-voter estimate. He could argue, instead, that the registered voters at the bottom of the likely-voter scale are just as likely to vote as those who score highest on the likely-voter scale. That would imply, for instance, that registered voters who say that they always vote, and those who say that they seldom vote, are equally likely to vote. Good luck with that. How else can he rationalize ignoring the likely voter estimates?
There TIA's data hold up somewhat better, although his probabilities don't. While the national polls (prior to TIA's massaging) fit the official results rather closely, the state polls do not fit as well. The median state poll in TIA's analysis, prior to TIA's allocation of undecideds, had a Kerry margin about 2.8 points larger than Kerry's actual performance. After allocating undecideds 75% to Kerry, the median discrepancy is about 4.5 points.
Looking at the crucial battleground states, the discrepancies don't seem very suspicious.
TruthIsAll had Kerry ahead by three points in Ohio, but only one Ohio poll out of the last ten actually put Kerry ahead. (This discrepancy could owe to flukes of timing, TIA's preference for registered-voter results, and/or TIA's reported propensity to ignore certain survey results he regards as biased.) Thus, although TIA's final model (again, the link -- find "OH") gave Kerry an 86.49% probability of winning Ohio, most observers would have leaned in the other direction. In my roundup, combining RealClearPolitics.com and electoral-vote.com figures, Bush had an average 1.9-point lead in the Ohio polls, quite close to his official margin of 2.1 points. (By the way, I checked two sources to ensure that RealClearPolitics' evident political bias didn't seep into its poll reporting; the results are closely comparable. Below, I link to RCP because its tables are easier to read.)
Similarly, because of TIA's allocation of most undecideds to Kerry, his Interactive Election Simulation spreadsheet gave Kerry a 73% chance of winning Florida despite depicting the race as nominally tied in the polls. In his final model report, Kerry has over an 86% probability of winning Florida. (The RealClearPolitics.com roundup gives Bush a narrow average margin of 0.6 points, with Bush ahead in 4 polls, Kerry ahead in 3, and the other 2 tied.) Bush did clearly do better in Florida than polls had projected, winning by just over 5 points.
Kerry won Pennsylvania as projected -- although even there, TIA's 96.69% probability in the final model seems misplaced. Kerry's 2.3-point winning margin in Pennsylvania was smaller than TIA's estimated 5 points prior to allocating undecideds. However, in the RealClearPolitics.com roundup, no poll put Kerry more than 4 points ahead; the average was 0.9 points, and the median was 2 points.
In summary, first of all, TruthIsAll's simulation results asserting a 99.8% probability that Kerry would win the electoral vote depended heavily on very favorable assumptions about Ohio and Florida. Second, the state polls may have overstated Kerry's official vote share, although part of the overstatement probably owes to TIA's selection of polls. (Later I will consider the argument that the state-level pre-election poll discrepancies and the exit poll discrepancies support each other as evidence of fraud.)
[1/6/07: TIA's response to this point is non-responsive, except to assert that I should report weighted means instead of medians -- as if that would somehow catapult Kerry into the lead in the Ohio polls?]
TIA and others have argued that the pre-election polls were biased against Kerry because they do not cover people who only use cell phones -- and these were disproportionately young voters who favored Kerry. Scott Keeter, drawing on a close analysis of Pew Research Center survey data, has concluded that because the polls are weighted to match the age distribution in the population, cell-phone-only voters had little effect on the polls' accuracy. "While cell-only voters were more supportive of John Kerry than voters overall, they were similar to voters within their own age cohort." (Keeter's findings also appeared in Public Opinion Quarterly.) Certainly Pew's age weightings seem fairly close to the mark. For instance, in their final pre-election poll (weighted), respondents aged 18-24 comprise 8.5% of the likely voters, and respondents 18-34 total 22.9%. In the Current Population Survey, self-reported voters 18-24 are 9.3% of all voters, and voters 18-34 are approximately 23.8% (derived from Table B). If one assumes, generously, that voters 18-24 favored Kerry about 60% to 40%, then any underrepresentation of young voters may have cost Kerry a small fraction of a percentage point in Pew's likely voter estimate.
[1/6/07: TIA's response to this point is non-responsive. These findings tend to contradict TIA's claim that the likely voter models were biased against Kerry, and there isn't much TIA can do about the fact.]
Previous: Introduction
Next: (2) The "Rules": Did They Favor Kerry?
Up to Table of Contents