A chart is a claim. A bar chart comparing two groups claims that one is larger than the other. A trend line claims the direction is up or down. A scatter plot with a superimposed regression line claims there is a relationship between two variables. None of those claims are self-evidently true from the visual alone: they depend on whether the underlying data meets the conditions required to support the claim. The statistics run before the chart are how those conditions get checked.
Skipping the pre-chart statistics does not make the chart neutral. It makes the claim unsupported, which is a different thing from being wrong. A chart built on data that violates the assumptions behind the analysis will be wrong for some data inputs and right for others, with no indication of which situation applies in the current case.
Step 1: understand the distribution
The first question about any numeric variable is: what does its distribution look like? Not to satisfy a normality requirement, but because the distribution determines which summary statistics are honest representations of the data and which ones mislead.
A mean reported for a log-normally distributed variable (revenue, session duration, latency) is pulled toward the high end by a small number of extreme values and does not represent what most observations look like. A median is more informative for that distribution. The summary statistic that appears in the chart should be the one that honestly represents the center of the data for that specific distribution.
import numpy as np
import pandas as pd
from scipy import stats
import matplotlib.pyplot as plt
# Visual check first: histogram + Q-Q plot
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))
ax1.hist(df['revenue'], bins=50)
ax1.set_title('Revenue distribution')
stats.probplot(df['revenue'], dist='norm', plot=ax2)
ax2.set_title('Q-Q plot (Normal)')
plt.tight_layout()
# Formal normality tests (use both; they have different power profiles)
stat_sw, p_sw = stats.shapiro(df['revenue'].sample(min(5000, len(df))))
stat_ks, p_ks = stats.kstest(
(df['revenue'] - df['revenue'].mean()) / df['revenue'].std(), 'norm'
)
print(f"Shapiro-Wilk p={p_sw:.4f}")
print(f"Kolmogorov-Smirnov p={p_ks:.4f}")The visual check (histogram and Q-Q plot) comes first. Formal normality tests are sensitive to sample size: with large samples, trivially small deviations from normality produce significant p-values. With small samples, they have low power to detect real non-normality. The visual is more informative than the p-value in most analytical contexts. The formal test confirms what the visual suggests; it does not replace judgment.
The practical consequence of non-normality is test selection. Many common statistical tests (t-test, ANOVA, Pearson correlation) assume normally distributed residuals or data. When that assumption is violated, non-parametric alternatives (Mann-Whitney U instead of t-test, Spearman instead of Pearson) or bootstrap methods are appropriate.
Step 2: identify and handle outliers deliberately
An outlier is an observation that is unusually far from the rest of the data. It might be a data quality error, a genuine extreme event, or a signal worth investigating rather than removing. The choice of how to handle it depends entirely on which of those it is, and that determination requires investigation, not a mechanical rule.
# IQR method: flag, not remove
Q1 = df['revenue'].quantile(0.25)
Q3 = df['revenue'].quantile(0.75)
IQR = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR
outliers = df[(df['revenue'] < lower) | (df['revenue'] > upper)]
print(f"Flagged {len(outliers)} outliers ({len(outliers)/len(df):.1%} of data)")
print(outliers[['customer_id', 'revenue', 'order_date']].head(10))I flag outliers and inspect them before deciding anything. A transaction with revenue of $847,000 in a dataset where the 99th percentile is $12,000 is worth looking at. It might be a B2B enterprise deal that is legitimately in the dataset and should stay. It might be a data entry error with a missing decimal point. It might be a refund recorded as a positive value. All three require different responses.
When an outlier is confirmed as valid, the right response is often to report the analysis with and without it, so the reader can see how much the outlier drives the result. A finding that only holds when one data point is excluded is a fragile finding, and the reader deserves to know that.
Step 3: test significance before declaring a difference
Two groups with different means in a sample do not necessarily have different means in the population. The difference could be sampling noise. Significance testing is the instrument that quantifies the probability of observing a difference this large or larger by chance alone if the two groups were actually identical in the population.
The test selection depends on the distribution check from Step 1 and on the study design:
| Situation | Parametric test | Non-parametric alternative |
|---|---|---|
| Two independent groups | Welch's t-test | Mann-Whitney U |
| Two paired groups (before/after) | Paired t-test | Wilcoxon signed-rank |
| Three or more groups | One-way ANOVA | Kruskal-Wallis |
| Relationship between two variables | Pearson correlation | Spearman correlation |
| Categorical association | Chi-square test | Fisher's exact (small samples) |
from scipy.stats import mannwhitneyu
group_a = df[df['segment'] == 'premium']['revenue']
group_b = df[df['segment'] == 'standard']['revenue']
stat, p = mannwhitneyu(group_a, group_b, alternative='two-sided')
print(f"Mann-Whitney U: statistic={stat:.0f}, p={p:.4f}")
# Effect size: common language effect size f (for Mann-Whitney)
n_a, n_b = len(group_a), len(group_b)
f = stat / (n_a * n_b)
print(f"Effect size (f): {f:.3f} (0.5 = no effect, 1.0 = complete separation)")Step 4: report effect size alongside p-value
A p-value below 0.05 does not mean the difference is large or practically meaningful. With a large enough sample, a 0.3% difference between two groups will be statistically significant. Reporting the p-value without the effect size tells the reader that the difference is unlikely to be noise but nothing about whether it is worth caring about.
Effect size metrics translate statistical significance into practical terms: Cohen's d for mean differences (small: 0.2, medium: 0.5, large: 0.8), Cliff's delta for non-parametric comparisons, Cramér's V for categorical associations. The combination of a statistically significant result and a small effect size is a common and important finding: it means the difference is real but may not be large enough to justify action.
The chart that goes into the report after these four steps is a different artifact from the chart drawn from raw data. It represents a claim that has been checked: the distribution was understood, the outliers were handled deliberately, the significance was tested with the right test for the data, and the effect size was reported alongside the p-value. That chain of work is what makes the chart worth reading.