An A/B test is an experiment. Like any experiment, its validity depends on decisions made before the data is collected: what effect size is meaningful enough to detect, how many observations are needed to detect it reliably, what the test statistic will be, and what significance threshold will govern the decision. A test designed after observing the data, or stopped when a significant result appears and restarted when it doesn't, is not an experiment. It is a procedure for finding significant results regardless of whether they exist.
This connects directly to the BA-04 project, where I analyzed a quasi-experiment on 95,824 Olist orders examining the effect of delivery SLA on customer satisfaction. The analysis required the same pre-analysis discipline as a prospective A/B test: defining the estimand, choosing the test, checking assumptions, and reporting not just what was significant but what the effect size implied for the business decision.
The minimum detectable effect: designing for what matters
The first question in A/B test design is not "how many users do I need?" It is "what is the smallest effect I care about?" The minimum detectable effect (MDE) is the smallest difference between treatment and control that, if real, would be worth acting on. It is a business judgment, not a statistical one.
If the intervention costs $50,000 to implement and the metric is conversion rate, an MDE of 0.1 percentage points (0.001) produces a sample size requirement in the millions and a test that takes months to run. An MDE of 2 percentage points produces a manageable sample size and a test calibrated to detect effects that actually justify the cost. Setting the MDE too small produces tests that are underpowered for what matters; setting it too large produces tests that would miss a real and meaningful effect.
from scipy import stats
import numpy as np
def sample_size_per_group(baseline_rate, mde, alpha=0.05, power=0.80):
"""
Two-proportion z-test sample size calculation.
baseline_rate: conversion rate in control (e.g. 0.03 for 3%)
mde: minimum detectable effect as absolute percentage points (e.g. 0.005 for 0.5pp)
"""
p1 = baseline_rate
p2 = baseline_rate + mde
pooled = (p1 + p2) / 2
z_alpha = stats.norm.ppf(1 - alpha / 2) # two-tailed
z_beta = stats.norm.ppf(power)
n = ((z_alpha * np.sqrt(2 * pooled * (1 - pooled)) +
z_beta * np.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2) / (mde ** 2)
return int(np.ceil(n))
# Example: baseline conversion 3%, MDE 0.5pp, 80% power
n = sample_size_per_group(baseline_rate=0.03, mde=0.005)
print(f"Required per group: {n:,}")
print(f"Total required: {2*n:,}")Statistical power: the probability of finding what is there
Statistical power is the probability that the test will detect a real effect of the specified size, given the sample size and significance threshold. A power of 0.80 means that if the true effect equals the MDE, the test has an 80% chance of producing a significant result. It also has a 20% chance of a false negative: failing to detect a real effect.
Most practitioners know about Type I errors (false positives, controlled by the significance threshold alpha) and fewer think about Type II errors (false negatives, controlled by power). An underpowered test is not a neutral instrument: it is systematically more likely to miss real effects than to detect them. When an underpowered test produces a non-significant result, the conclusion "the treatment had no effect" is unjustified. The correct conclusion is "the test was not sensitive enough to detect an effect of this size."
| Power level | False negative rate | Practical implication |
|---|---|---|
| 0.50 | 50% | Coin flip for detecting real effects: essentially uninformative |
| 0.80 | 20% | Industry standard minimum for most business tests |
| 0.90 | 10% | Appropriate when false negatives are costly (safety, medical, high-stakes product) |
| 0.95 | 5% | Matches Type I error rate at alpha=0.05; symmetric error budget |
The conditions under which a significant result is still wrong
A p-value below 0.05 does not mean the experiment was valid. Several conditions can produce a significant result that should not be acted on.
Peeking and early stopping. Looking at the results while the test is running and stopping when significance is reached inflates the Type I error rate substantially. If you check at ten points during the experiment and stop when p < 0.05 first appears, your actual false positive rate can exceed 30%, not 5%. The correct approach is to fix the sample size before starting and not look at the results until the test is complete. If continuous monitoring is required, use sequential testing methods (alpha spending functions, always-valid p-values) that are designed for it.
Multiple comparisons without correction. Testing five metrics simultaneously at alpha = 0.05 gives each metric a 5% chance of a false positive, but the probability that at least one of the five produces a false positive is about 23%. When multiple metrics are tested, the significance threshold must be adjusted: Bonferroni correction (alpha / number of tests) is conservative but simple; Benjamini-Hochberg controls the false discovery rate at a less conservative level.
Sample Ratio Mismatch. The assignment ratio between treatment and control should match the designed ratio throughout the experiment. If 50% of users were supposed to be in treatment and only 43% ended up there, the randomization was compromised: some mechanism is differentially affecting which users enter which group. A significant result from a test with a sample ratio mismatch is uninterpretable.
# Sample Ratio Mismatch check
n_control = 48_204
n_treatment = 41_977
total = n_control + n_treatment
expected_treatment = 0.50 # 50/50 split
observed_ratio = n_treatment / total
print(f"Observed treatment ratio: {observed_ratio:.3f} (expected {expected_treatment:.3f})")
# Chi-square test for SRM
chi2 = (n_control - total * (1 - expected_treatment))**2 / (total * (1 - expected_treatment)) + \
(n_treatment - total * expected_treatment)**2 / (total * expected_treatment)
p_srm = 1 - stats.chi2.cdf(chi2, df=1)
print(f"SRM test p-value: {p_srm:.4f} {'SRM DETECTED' if p_srm < 0.01 else 'OK'}")Novelty effects. Users respond differently to new things. A new checkout flow may increase conversion in the first week simply because it is different, not because it is better. If the experiment ran for only one week, the effect may not persist. Run tests long enough to cover at least one full user behavior cycle (typically one week minimum, two if there are weekend effects).
Reporting an A/B test result honestly
A complete A/B test report includes: the pre-registered hypothesis and MDE, the designed sample size and achieved sample size, the SRM check result, the effect size (not just the p-value), the confidence interval on the effect, the power post-hoc if the result was not significant, and the business recommendation with its assumptions stated explicitly.
A report that shows only "treatment: 3.4%, control: 3.1%, p=0.03, significant" has omitted everything that would let a reader assess whether to trust the result. The 0.3 percentage point difference with a wide confidence interval that nearly includes zero, from a test that ran three days before reaching the planned sample size, from a product with high week-over-week volatility: those details determine whether the result is actionable or whether it requires a replication. The numbers alone never do.