Li, E. Y., & Rhoads, C.
Evaluating methods for handling multilevel selection for the purpose of generalizing cluster randomized trials
Over the past decade, the generalizability of randomized experiments, defined as the level of consistency between an estimated treatment effect in a nonrandom sample and the true treatment effect in a target population, has received increasing attention from the research community. Existing methods focus on either: (a) prospectively preventing or (b) retrospectively adjusting away, the bias caused by the nonrandom selection of institutions, such as schools, into a study sample. Existing methods overlook the multilevel nature of the selection process that occurs when institutions volunteer for a research study. This study explores methods to adjust away bias caused by this multilevel selection process. A simulation study evaluates the bias reducing properties of different methods for estimating and utilizing inverse probability of participation (IPP) weights to reduce bias. Methods that incorporate both student and school IPP weights reduce more bias than methods that only incorporate the school IPP weights.
Testing, Psychometrics, Methodology in Applied Psychology, 2020, Vol. 27, pp. 453-476, DOI: 10.4473/TPM27.3.8
