Review of: Paul J. Lavrakas, Michael W. Traugott, Courtney Kennedy, Allyson L. Holbrook, Edith D. de Leeuw, and Brady T. West, eds. Experimental Methods in Survey Research: Techniques That Combine Random Sampling with Random Assignment
I wholeheartedly agree that nearly any survey presents a valuable opportunity to conduct survey experiments. That said, I’d like to insert a plea that we move on from validity'' framework when discussing research designs. In my view, the classic internal versus external validity distinction for survey experiments is mostly not helpful. We should focus on estimands, not alternative flavors of validity. For survey experiments conducted with online convenience samples, a common estimand is the Sample Average Treatment Effect, or SATE. If the survey experiment is designed and analyzed well, then estimates of the SATE will, on average, be close to the SATE. This isinternal validity,’’ or a claim about the unbiasedness or consistency of the full set of procedures that lead to an estimate. Of course, a SATE on an online convenience sample might not be the same as a SATE in a probability sample of Minnesotans. A SATE estimate is said to lack external validity when it is not equal to a particular Population Average Treatment Effect (PATE). There are many possible populations, so there are many possible PATEs: the ATE among all Americans in 2020 or the ATE among world citizens in 1983. Since these PATEs could easily be different, any SATE will always be ``externally invalid’’ for some PATE or other, regardless of whether the subjects are drawn at random from a well-defined population or not. By this logic, no design can ever achieve external validity because there may always be some new population to which the results will fail to generalize. Any claim about a study’s external validity would be sharper if the target estimand were made explicit.