A/B Testing Statistical Significance (p-value), Bayesian Win Probability & Sample Size Calculator (2026)

Given Control conversion rate $p_A = c_A / n_A$ and Variant rate $p_B = c_B / n_B$, the pooled proportion under the null hypothesis ($p_A = p_B$) is $\hat{p} = (c_A + c_B) / (n_A + n_B)$. The pooled standard error is $SE = \sqrt{\hat{p}(1 - \hat{p})(1/n_A + 1/n_B)}$, and the test statistic is $Z = (p_B - p_A) / SE$. The two-tailed $p$-value is $2 \times (1 - \Phi(|Z|))$, where $\Phi$ is the standard normal cumulative distribution function.

If your experiment is configured for a 50/50 traffic split across 20,000 users, you expect roughly 10,000 users in Control A and 10,000 in Variant B. If Control receives 10,450 and Variant receives 9,550, a $\chi^2$ test yields $p < 0.001$—indicating a **Sample Ratio Mismatch**. SRM usually means slow variant page load, client-side redirect bugs, or bot filtering dropped a specific segment of users from one bucket, rendering the conversion comparison biased.

Standard fixed-horizon Z-tests assume you evaluate the $p$-value once after reaching your pre-calculated sample size. Checking a test every day for 14 days and stopping the first moment $p$ dips below 0.05 inflates your true false-positive (Type I error) rate from 5% to over **25%–30%**. Always commit to the MDE sample size upfront.

If your baseline conversion rate is **5.0%**, a **10% Relative MDE** means detecting a shift to **5.5%** ($5.0\% \times 1.10$), whereas a **10% Absolute MDE** would mean jumping from 5.0% to **15.0%**. Most product and growth teams specify Minimum Detectable Effect in relative terms (e.g., 5% to 15% relative lift).

A Two-Tailed test splits your $\alpha$ error budget equally between both directions ($Z_{\text{crit}} = \pm 1.96$ at 95% confidence), allowing you to rigorously detect both whether Variant B is significantly better OR significantly worse than Control A. Industry experimentation platforms (Optimizely, VWO, GrowthBook, Statsig) recommend Two-Tailed tests by default.