« So If We Agree on All This, How Did We Wind Up with Bush? |
Main
| Voters on Bush’s Social Security Plan: Thanks, but No Thanks »
January 25, 2005
That Cleared Up Everything, Right?
Edison/Mitofsky released their massively-detailed evaluation of the 2004 NEP exit poll last week. As Mark Blumental of Mystery Pollster observes, there is much useful data in the report, even if much of it is presented in fairly arcane fashion.
Among other things we learn the following:
1. There was, in fact, more error in this exit poll than in previous exit polls and data are presented on WPE (within-precinct error) that pretty much clinch the case. The increased level of error meant that, after the exit poll data were initially processed and corrected, they had to be further weighted, to a greater degree than in previous exit polls, to correspond to the actual election outcome. Hence the Kerry 51-48 popular vote win in the initial exit poll data that became a 51-48 popular vote defeat upon final weighting.
2. There seem to have been serious problems with inexperienced and poorly-trained interviewers, who were not able to administer the questionnaires in the fashion intended by Edison/Mitofsky.
3. There seem to have been serious sampling problems ("clustering effects") with the national exit poll that, for example, contributed to the 44 percent support figure for Bush among Hispanic respondents, which Edison/Mitofsky appears to be backing away from.
The problem with all these admissions and the copious data in the Edison/Mitofsky report is that nowhere to they really make much of a stab at answering the underlying questions that are raised by the report's findings.
1. If there was higher error than normal, why was that? What makes this election different and will the next election be even more different? If Kerry voters were more willing to be interviewed (or Bush voters less willing), which appears to be Edison/Mitofksy's primary rationale for the high error level, why was this and why would this kind of behavior be particularly common in this election? And how does this square with the lack of variation in response rates across precincts that voted heavily for Kerry, heavily for Bush and inbetween? (See Mystery Pollster's good discussion of this issue.)
2. If so many interviewers did so poorly, why was that? Did Edison/Mitofsky do an exceptionally poor job of selecting and training interviewers and, if so, why was that? Or is exit polling just getting harder to do, so interviewers that might have been adequate 4-8 years ago are not adequate today?
3. If the sampling problems were so serious, why was that? Were they worse than in previous years and, if so, why? Did Edison/Mitofsky do a poor job of dealing with these problems or was there nothing they could have done? Should we have been using aggregated state data instead of the national data for relatively small demographic groups all along like they imply we should with this year's data? Or, again, is there something about the way the country's changing that's making always-exising sampling problems of the exit polls worse?
I don't know the answers to these questions. But I think they deserve answers, hopefully sooner rather than later, while there's still time to restore faith in an exit polling system that is now faced with widespread skepticism.
Posted by Ruy Teixeira at 06:18 PM | link
|