Coppock, Alexander, Thomas J. Leeper, and Kevin J. Mullinix. “The Generalizability of Heterogeneous Treatment Effect Estimates Across Samples” Working Paper.

Abstract

Abstract: The extent to which studies conducted with non-representative convenience samples are generalizable to broader populations depends critically on the level of treatment effect heterogeneity. Recent inquiries (e.g., Mullinix et al. 2015; Coppock forthcoming) have found a strong correspondence between average treatment effects estimated in nationally-representative experiments and in replication studies conducted with convenience samples. In this paper, we consider three possible explanations: low levels of effect heterogeneity, high levels of effect heterogeneity that are unrelated to selection into the convenience sample, or just good luck. We reanalyze 26 original-replication study pairs (encompassing 98,425 individual survey responses) to assess the extent to which a model of heterogeneity in treatment response estimated on the original dataset predicts the heterogeneity in the replication experiment, and vice-versa. While there are exceptions, the overwhelming pattern that emerges is one of treatment effect homogeneity, providing a partial explanation for strong correspondence across both unconditional and conditional average treatment effect estimates.

Figure

Figure 1 from paper, showing correlations of conditional average treatment effect estimates across original and MTurk replications.

Coppock, Leeper, and Mullinix 2017 Figure 1

Bibtex citation

@article{Coppockgeneralizability,
    Author = {Coppock, Alexander and Leeper, Thomas J. and Mullinix, Kevin J.},
    Title = {The Generalizability of Heterogeneous Treatment Effect Estimates Across Samples},
    Year = {2017},
    Journal = {Unpublished manuscript}
}