Psychology Methods Hour: Power Analysis Has Lost Its Way: New Methods To Bring It Back Home

In a time when the alarms of research replicability are sounding louder than ever, mapping out studies with statistical and inferential integrity is of paramount importance. Indeed, funding agencies almost always require grant applicants to present compelling a priori power analyses to justify proposed sample sizes, in an effort to ensure a sound investment. Unfortunately, even researchers’ most sincere attempts at sample size planning are fraught with the fundamental challenge of setting numerical values not just for the focal parameters for which statistical tests are planned, but for each of the model’s other more peripheral or contextual parameters as well. As we plainly demonstrate, regarding the latter parameters, even in very simple models well-intentioned numerical guesses that are even slightly off can undermine power for the assessment of the more focal parameters that are of key theoretical interest. Toward remedying this all-too-common but seemingly underestimated problem in power analysis, we adopt a hope-for-the-best-but-plan-for-the-worst mindset and present new methods that attempt (1) to restore appropriate conservatism and robustness, and in turn credibility, to the sample size planning process, and (2) to greatly simplify that process. Derivations and suggestions for practice are presented using the framework of measured variable path analysis models as they subsume many of the types of models (e.g., multiple linear regression, ANOVA) for which sample size planning is of interest.