A/B Test Analysis — Conversion with Statistical Rigor

Context

A product team tested a new version of the checkout page (variant B) against the current version (variant A), aiming to lift the conversion rate. The two groups will always produce two different numbers — some difference is guaranteed by chance alone. So leadership is left with the question the raw numbers cannot settle on their own: does the change actually work, or is the observed difference just statistical noise?

This project is the analysis behind that decision. Shipping a variant that does nothing costs engineering time and quietly erodes trust in experimentation; killing a variant that does work leaves revenue on the table. The deliverable is not a chart — it is a call that holds up when someone pushes back on it.

Business Problem

Is variant B's conversion rate statistically higher than variant A's — enough to justify a full rollout?

Everything else in the project serves that one sentence. The answer has to arrive with three things attached: the size of the effect, the uncertainty around it, and whether the test collected enough data to be trusted in the first place. A recommendation missing any of those is a guess wearing a percentage sign.

Architecture

Analysis pipeline

Deliberately small. This is an analytics problem, not an engineering one, and a notebook that runs top to bottom on a fresh kernel is a more honest artifact here than a pipeline would be. SQLite earns its place by keeping the group-level queries readable and reproducible instead of buried inside a chain of dataframe operations.

Experimentation flow

The hypotheses are written down before the test is run, and the decision rule is fixed before the p-value is known. That ordering is the entire discipline — it is what separates an experiment from a search for a number that agrees with you.

Data Source

• Source: a public A/B testing dataset (the "A/B Testing" dataset on Kaggle). An alternative track uses a crypto event study built from real market data via the CoinGecko API — comparing returns before and after an event window.
• Storage: CSV for the raw extract, SQLite for querying and reproducibility.
• Size: thousands of sessions/users, split between a control group and a treatment group.
• Key variables: group (A/B), conversion (yes/no), time on page, device.

Known limitations, stated up front: there is no segmentation by acquisition channel, so a mix shift between groups would be invisible in this data. The test window is also possibly too short to capture seasonality — a weekday-only window is not a claim about weekends. Both constraints get carried into the final recommendation rather than quietly dropped.

Methodology

Framework: CRISP-DM — Business Understanding → Data Understanding → Data Preparation → Modeling (here, hypothesis testing) → Evaluation → Deployment (the recommendation).

1. Business & data understanding — define the success metric (conversion rate) and validate the experiment design: what is control, what is treatment.
2. Data preparation — check group balance (similar sizes and characteristics), remove users appearing in both groups, confirm a consistent test window.
3. Descriptive analysis — compute the observed conversion rate per group and visualize it with confidence intervals, so the overlap is visible before any test is run.
4. Hypothesis test — state H0 (no difference) and H1 (B ≠ A) explicitly; run a Z-test for two proportions (or a chi-square test) and justify the choice rather than defaulting to it.
5. p-value, interpreted correctly — report what the p-value is and, just as importantly, what it is not.
6. Confidence interval of the difference — quantify the effect and its uncertainty, not just a binary "significant / not significant."
7. Power analysis — was the sample large enough to detect the expected effect? Compute the minimum required sample size and state whether the test was adequately powered.

AI-assisted communication (AI assists, human validates)

Once the statistics are computed and validated by hand, an LLM translates the technical result — p-value, confidence interval, power — into plain, stakeholder-friendly language. The human owns and validates every number and every claim; the AI only rephrases a conclusion that was already verified. It is a writing step, not an analysis step, and it never touches the math.

Challenges

1. Interpreting the p-value correctly

The p-value is the most misquoted number in analytics. It is not the probability that H0 is true, not the probability the variant works, and not a measure of how big the effect is. It is the probability of observing a difference at least this extreme if there were no real difference at all. A p < 0.05 that gets narrated as "we're 95% sure B is better" is a wrong sentence attached to a right calculation — and it is the sentence, not the calculation, that leadership acts on. The analysis reports what the p-value is and what it is not, and pairs it with the confidence interval so the decision rests on effect size and uncertainty rather than a threshold crossing.

2. Getting enough sample — and knowing when there isn't

A non-significant result has two very different meanings: "there is no effect" or "there wasn't enough data to see one." Treating the second as the first kills good variants. Power analysis is what tells them apart, which is why the minimum detectable effect and the required sample size are computed up front instead of reverse-engineered afterward. If the test turns out to be underpowered, the honest recommendation is extend the test — not don't ship.

3. Avoiding p-hacking across segments

Segment analysis is where good intentions go to die. Slice by device, then by channel, then by new vs. returning, and at some point one slice crosses p < 0.05 purely by chance — that is what a 5% false-positive rate means when you run twenty comparisons. The guard is procedural, not statistical: the primary metric and primary test are locked before looking, and segment findings are labeled as exploratory — hypothesis-generating input for the next test, with the multiple-comparisons caveat documented rather than buried.

4. Trusting the design before trusting the numbers

The statistics are only as good as the randomization underneath them. If the groups are badly unbalanced, if users appear in both arms, or if the window shifted mid-test, the test is invalid no matter how clean the output looks — and none of that announces itself in the p-value. The validation checks come first, and if the split is broken, the finding is that the experiment can't be read, which is a real and reportable result.

Results

Planned outputs:

• The observed difference between the groups, and whether it is statistically significant (p-value).
• The confidence interval of the conversion difference — the effect and its uncertainty.
• The power the test actually achieved, against the effect it was designed to detect.
• Segments (e.g., mobile vs. desktop) where the effect looks stronger or weaker, flagged as exploratory.
• A clear final call — ship, don't ship, or extend the test — backed by the statistical evidence and summarized in one sentence for a non-technical audience.

Tech Stack