Matthew Graham and Alexander Coppock (2019), “The Perils of Self-Assessed Attitude Change.” Working paper.

Abstract

Surveys often ask respondents to self-report how events or information changed their attitudes. Does [event X] make you more or less likely to vote for a politician? How did [information X] affect your opinion? Would you be more supportive of a candidate who took [position X]? We show that the self-assesment question type exhibits poor measurement properties. Using 18 mini-experiments across three studies, we compare this question type—and eight alternative ways of asking subjects to assess counterfactuals—to randomized experiments. When asked to report how their attitudes change, subjects appear to frequently overestimate the magnitude of treatment effects and sometimes get the sign wrong. Our results are broadly consistent with the notion that respondents misreport their attitude change because they engage in response substitution. Self-reports of attitude change appear to be infected by respondents’ absolute level of support for the candidate or policy in question.

Figure

Figure 2 from paper, showing the effect of alternative question formats on reporting change by partisanship

Graham and Coppock Figure 2