By Scott Winship

Garance Franke-Ruta has posted a response to Ruy's and my critiques of the American Environics data presented in her recent piece in the American Prospect. We challenged the American Environics findings that Americans were becoming less egalitarian on gender issues and less civically engaged. The most important argument she makes is that by relying on the National Election Study, we are primarily looking at voters, while American Environics surveys voters and nonvoters alike. To be fair to Franke-Ruta, this is the explanation given by American Environics rather than her own. But their excuse is really embarrassingly easy to knock down.

If it wasn't obvious from our earlier posts, both Ruy and I tabulated results for all adults rather than just voters. The NES is a nationally representative sample of American households, restricted to adults. It is true - as Ruy has consistently noted for years - that the NES tends to exaggerate how many people vote (and how many vote for the winner), but that is a problem of response bias rather than a problem with the underlying design of the survey. Some people don't want to admit to not voting. But that does not change the fact that comparing the NES aggregate results to the American Environics aggregate results is a valid, apples-to-apples comparison. So we could stop right there and render American Environics's critique moot.

But let's not. American Environics claims that voters and nonvoters have markedly different levels and trends in their data. To check this claim, I re-ran the NES figures separately for voters and nonvoters. The fact that the NES overstates voting means that comparing "voters" to "nonvoters" in the NES will understate differences between the two groups, but if both groups show similar trends and levels and differ from the American Environics data, the response bias is irrelevant to this controversy.

The following tables compare the American Environics figures (for all adults) to the NES figures for all adults, voters, and nonvoters. As Franke-Ruta notes (and as we noted ourselves) the questions are not exactly comparable but get at the same underlying dimensions and should show similar trends. Note that we subtract American Environics's gender figures from the article from 100 so that they represent gender egalitarianism rather than inegalitarianism.

To summarize, the NES indicates:

* More gender egalitarianism (on an admittedly different question) than the American Environics research for both voters and nonvoters
* Increases in gender egalitarianism since 1992 among both voters and nonvoters (as opposed to the decline American Environics claims)
* More civic engagement (on an admittedly different question - though not much different) than American Environics finds for all adults, for voters, and (in 2004) for nonvoters.
* Only a small, statistically insignificant change in civic engagement since 1992 (as opposed to the huge decline claimed by American Environics).

American Environics could claim that the NES also suffers from response bias on the gender egalitarianism and civic engagement questions, though that's not the actual criticism they initially made. It is also the case that their own data could be subject to various biases. The fact that they have 12 years of experience with their survey while the NES has been conducted for 56 years points toward the likely superiority of the NES data.

More to the point, if response bias is constant across time, then the trends in the NES will be unaffected, and the trends in the NES are dramatically different than those in the American Environics data. Unless American Environics has a story about why response bias would have increased in 2004 relative to 1992 - and increased a lot - this explanation doesn't have legs. (Remember too that, looking back to 1984, the NES indicates that just 67 percent of adults were civically engaged by our definition. Are we really to believe that increasing response bias explains the big increase in civic engagement the NES shows between 1984 and 1992 too?)

And since we generally show the same discrepancies between the American Environics data and NES voters and nonvoters, any response bias would have to be independent of the response bias that causes some nonvoters to say they are voters.

For all these reasons, the basic critique of the American Environics data still stands, as does the need to cast the data net widely when seeking to understand value change. Relying on the Environics data alone is clearly not a wise approach.