I have pilot data. What is my power?
Source:vignettes/from-pilot-find-power.Rmd
from-pilot-find-power.RmdSuppose we ran a pilot study and know our planned sample size for the full study. We have a specific treatment effect in mind. We want to find the power—the probability that the study will produce a statistically significant result if the true effect equals our assumed value.
from_pilot() piped into find_power()
answers this question.
Example
Cutler, Pietryka, and Rainey (2024) run a pilot replication of Ahler and Sood (2018), who find that correcting respondents’ misperceptions of their out-party reduces affective polarization on a 101-point feeling thermometer. Ahler and Sood estimate a treatment effect of 6.4 points with a 95% confidence interval of [3, 10]. Cutler et al. use the pre-post measurement strategy of Clifford, Sheagley, and Piston (2021), measuring feelings toward supporters of the out-party before and after treatment. In the pilot, with 85 per condition, they estimate a standard error of 2.13.
Suppose we plan to run the full study with 500 per condition. The lower bound of Ahler and Sood’s 95% confidence interval is 3 points. We use this as our assumed effect, tau = 3.
library(powerrules)
from_pilot(se_pilot = 2.13, n_pilot = 85) |>
find_power(n_planned = 500, tau = 3)#> -- Power Analysis ------------------------------------------------------
#> Design: balanced, between-subjects
#> Source: pilot data (conservative)
#> CI level: 90% (size-0.05 test of directional hypothesis)
#>
#> Inputs:
#> SE (pilot) = 2.13
#> n (pilot) = 85 per condition
#> n (planned) = 500 per condition (1,000 total)
#> tau = 3
#>
#> Predicted SE = sqrt(85 / 500) * (sqrt(1/85) + 1) * 2.13 = 0.97 [Rule 9]
#> tau / SE = 3 / 0.97 = 3.08
#> Power = 1 - pnorm(1.64 - 3.08) = 92% [Rule 2]
#>
#> -- Manuscript sentence (edit as needed) --------------------------------
#> For a balanced, between-subjects design with 500 respondents per
#> condition (1,000 total), using pilot data with a standard error of
#> 2.13 (85 per condition) and a conservative adjustment for pilot noise,
#> the predicted standard error is 0.97. Using a one-sided test at the
#> 0.05 level, the experiment has 92% power to detect a treatment effect
#> of 3 units.The output predicts the standard error using the pilot SE, sample size, and a conservative adjustment factor (Rule 9 from Rainey (2026)), then computes power for the assumed effect (Rule 2). With 500 per condition, the study has 92% power to detect a treatment effect of 3 points.
The conservative adjustment factor
Standard errors estimated from small pilot studies are noisy. If the pilot SE happens to be smaller than the true SE, a naive rescaling would predict a standard error that is too small, overstating the study’s power (Albers and Lakens 2018; Rainey 2026). The conservative adjustment factor inflates the predicted SE to protect against this risk. Despite the conservative adjustment, the study still has 92% power for tau = 3, suggesting the planned sample size is adequate.