By Alan Abramowitz

Jeff Jones of Gallup recently posted this reply to critics on their editors' blog:

Some consumers of the polls (including the Gallup Poll) have questioned poll results because party identification and other characteristics do not match the 2000 exit poll data. There are very good reasons why they may not match the exit poll data.

First, some treat the exit poll as a census. It is not a census, it is a survey based on sampling of voting precincts. There is a reason it is called an "exit poll" and not an "exit census." That's because it is a poll, and as such is subject to sampling variation and other polling errors just as any other poll would be. In fact, because exit polling relies on quota sampling (hand selection of survey spots according to population size and other factors), it has a higher degree of potential error than do the random samples on which telephone surveys are based. Thus, there is no basis on which to believe the exit poll numbers are in any way more accurate than any other number you get from a poll. They are all estimates. The one advantage of the exit poll is that they know everyone they interview is a voter, while pre-election polls rely on models to determine who is likely to vote and who is not. However, that does not mean their estimates are necessarily better, and they are definitely not error-free estimates of the electorate as many treat them. In fact, when multiple exit polls existed in the past, they very routinely differed in their estimates of the vote as well as their estimates of the demographic characteristics of the electorate. Even today the Los Angeles Times exit poll differs from the larger exit poll used by the networks.

Second, the exit poll measure of political party ID is fundamentally different from ours. We know that survey results can differ depending on how the data are collected. Our questions are read and responses obtained verbally over the phone. Their responses are obtained in self-administered questionnaires that present the questions in a visual format. Most survey research experts would be extremely cautious in comparing data obtained by a telephone interview versus that obtained in a self-administered paper-and-pencil questionnaire. That is in addition to question wording differences in the party ID question that can also have an effect on the results.

Third, a lot has changed since 2000. In the post-9/11 environment, terrorism has become one of the chief problems for government to deal with. The Republican Party has a large perceptual advantage on the terrorism issue. To assume that everything is as it was four years ago is a very risky assumption. While it is possible that in the end things could change once again so that partisanship looks much like it did in 2000, that is by no means certain or even likely.

My reply:

Mr. Jones's comments are inaccurate and misleading. First of all, yes, the national exit poll is based on a sample. But it's a huge sample--over 13,000 respondents in 2000. Because of its size, and because, as Jones acknowledges, the exit poll includes only individuals who have definitely voted, the margin of error should be much smaller than with pre-election telephone polls. Second, there is no reason to believe that a self-administered questionnaire would produce significantly different results from a telephone survey for an attitude as basic as party identification as long as the question asked is worded identically. The problem with the Gallup party ID question, as I have indicated elsewhere, is that it is a poorly worded question. Because of its lead-in, "in politics today," the Gallup question, in contrast to the question used by the National Election Studies, the CBS/New York Times Poll, and the national exit poll, measures a combination of current political preferences along with long-term partisan commitment. Finally, and this is most important, there is simply no reason to believe that the distribution of party identification in the American electorate has changed significantly since the 2000 presidential election.

Indeed, an examination of national exit polls and CBS/New York Times polls conducted since 1992 shows that there has been no significant change in party identification for the past 12 years despite wars, recessions, and quite variable election results. Even in the immediate aftermath of the September 11th attacks, the CBS/New York Times polls found no noticeable change in the distribution of party identification. The distribution of party identification in CBS/New York Times polls this year, approximately a four point Democratic advantage, is almost identical to the average for the past 12 years. It is therefore highly unlikely that the distribution of party identification in this year's national exit poll will differ substantially from that in the previous four national exit polls, all of which showed a Democratic advantage of between
3 and 5 points.

Unfortunately, Mr. Jones's comments are typical of the head-in-the-sand attitude that the Gallup Poll has displayed in recent years in response to any criticism of their work. During the 2000 campaign, we would remind readers, the Gallup tracking poll was the laughingstock of the polling community as its likely voter results gyrated wildly from week to week and sometimes from day to day, producing a ludicrous estimate of a 13 point lead for George Bush on October 26th. Although Gallup quietly abandoned their tracking poll this year, they continue to display the same arrogance and insensitivity to criticism that we saw then. Evidently party identification is not the only attitude that is impervious to change.