The Generalizability of Online Experiments Conducted During The COVID-19 Pandemic
The COVID-19 pandemic imposed new constraints on empirical research, and online data collection by social scientists increased. Generalizing from experiments conducted during this period of persistent crisis may be challenging due to changes in how participants respond to treatments or the composition of online samples. We investigate the generalizability of COVID era survey experiments with 33 replications of 12 pre-pandemic designs, fielded across 13 quota samples of Americans between March and July 2020. We find strong evidence that pre-pandemic experiments replicate in terms of sign and significance, but at somewhat reduced magnitudes. Indirect evidence suggests an increased share of inattentive subjects on online platforms during this period, which may have contributed to smaller estimated treatment effects. Overall, we conclude that the pandemic does not pose a fundamental threat to the generalizability of online experiments to other time periods.
PDF Appendix 1 Appendix 2 Appendix 3 Preprint 1 Preprint 2 Journal Replication archive (Harvard) Replication archive (Yale) Active maintenance Corrigendum Corrigendum in the journal BibTeX
Corrected in 2023. Running computational reproducibility checks, the Yale ISPS Data Archive found errors in the code that adjusted p-values for multiple comparisons. They made the article report the number of statistically significant differences inconsistently before and after adjustment, affecting Figures 2 and 3 and one paragraph on page 6. The corrected figures and text are in the corrigendum, linked above. All point estimates, standard errors and substantive conclusions are unchanged.
Peyton, Kyle, Gregory A. Huber and Alexander Coppock. 2023. The Generalizability of Online Experiments Conducted During The COVID-19 Pandemic: CORRIGENDUM. Journal of Experimental Political Science 10(2): 306–309. doi:10.1017/XPS.2022.23
