Our twitter feeds lit up with these two contradictory tweets about how vaccine attitudes changed as a result of the decision to pause administration of the Johnson & Johnson vaccine:

We criticized the poll question from @benwakana46 because it it uses the “change format.” Respondents are asked to evaluate how an event changed their attitudes – in this, did the announcement make them “more confident,” or did it make them “less confident.”

We wrote a paper showing that this kind of question is biased (https://m-graham.com/papers/GrahamCoppock_aaac.pdf). The main reason it’s biased is because of “response substitution.” Subjects just want to express that they “remain confident” in the vaccines, so they have to substitute in the only available response option, which in this case is “more confident.”

The poll from YouGov is much more sensible. It asks about attitudes using the “level format.” In the level format, subjects just report the level of the attitudes, i.e., whether they think vaccines are safe or not. YouGov polls people constantly, so they can generate estimates of the attitude effects of “surprise” events like the J&J pause with a simple before / after comparison.

If you are running a poll, but don’t happen to be in the field with exactly the right set of level questions before and after unforeseen events (i.e., the usual state of affairs), you might want to learn about attitude change using our counterfactual format. Here’s how we would have modified the polling question @benwakana46 tweeted into two polling questions:

Q1. The FDA has given emergency authorization for three COVID-19 vaccines in the US, including a vaccine by Johnson and Johnson that provides protection from COVID-19 in a single dose. So far, 6.5 million doeses of this vaccine have been administered. In response to reports that six women who received the Johnson and Johnson vaccine have had rare blood clots, the FDA and CDC recommended a pause in the administration of the vaccine. 

In light of this information, how condident in the vaccines are you?

1. Not confident at all 
2. Slightly confident
3. Somewhat confident
4. Fairly condident
5. Completely confident
Q2. Now imagine that we had asked you this question before the annoucement of the pause. How would you have answered the question, how condident in the vaccines are you?

1. Not confident at all 
2. Slightly confident
3. Somewhat confident
4. Fairly condident
5. Completely confident

An average of the Q1 - Q2 differences is an estimate of the average treatment effect of the announcement and the distribution of “effects” Q1-Q2 can be plotted or summarized like any continuous variable.

We think pollsters should immediately abandon change questions. They aren’t informative as estimates of attitude change because they are biased by response substitution. The counterfactual format works because it avoids response substitution. It directly asks about the levels of opinion in two counterfactual scenarios that repondents are able to imagine.

Developing counterfactual format questions is new and we hope there will be lots of variations on the wording we offer here (and the many other examples we describe in our paper). We welcome people developing counterfactual format questions to get in touch.