Exploratory data analysis without a structure produces one of two failure modes. The first is analysis that is thorough but unrepeatable: a notebook full of cells run in an order nobody documented, with conclusions that cannot be reproduced by anyone who opens the file later. The second is analysis that is organized but shallow: a template so rigid it goes through the motions without asking the questions the specific dataset demands.
The template I describe here sits between those two failure modes. It is structured enough to be reproducible and to ensure no critical category of check is skipped. It is flexible enough to be extended when the specific dataset raises specific questions. The structure is the floor, not the ceiling.
This template connects directly to the work in the BA-04 A/B test project and the DA-02 dashboard project, where the exploration phase determined the analytical approach before any visualization was produced.
Section 0: environment and data load
The first cell of every notebook sets up the environment deterministically: imports pinned to specific versions, random seeds set, display options configured, and the data loaded from a path that a fresh environment can resolve. A notebook that only runs on one person's machine because it depends on a relative path, an environment variable that was never documented, or a package version that was never pinned is not reproducible.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
import warnings
# Reproducibility
np.random.seed(42)
warnings.filterwarnings('ignore', category=FutureWarning)
pd.set_option('display.max_columns', 50)
pd.set_option('display.float_format', '{:.4f}'.format)
# Data load -- path relative to project root, not analyst's home directory
DATA_PATH = "../data/raw/olist_orders.parquet"
df = pd.read_parquet(DATA_PATH)
print(f"Loaded: {df.shape[0]:,} rows x {df.shape[1]} columns")Section 1: shape and schema
Before reading a single value, I need to know what the dataset looks like at the structural level: how many rows, how many columns, what types are they, and do the types match what the column names suggest?
# Shape, dtypes, memory
print(df.dtypes.value_counts())
print(f"Memory: {df.memory_usage(deep=True).sum() / 1e6:.1f} MB")
# Type mismatches: date columns stored as object?
date_cols = [c for c in df.columns if 'date' in c or 'at' in c or 'time' in c]
print("\nDate-like columns and their actual types:")
print(df[date_cols].dtypes)A column named order_date stored as object rather than datetime64 is a data loading issue that will produce wrong results in any time-based analysis without announcing itself. Finding it in Section 1 means fixing it before it propagates.
Section 2: completeness
Missing value rates, sorted descending. Any column with more than 5% nulls gets a note explaining whether the missing values are expected (an optional field), unexpected (a data quality issue), or informative (a null in "delivered_at" means the order was not delivered). The interpretation of a null depends on the domain, and the domain knowledge needs to be in the notebook, not in someone's head.
null_summary = (
df.isnull()
.mean()
.rename('null_rate')
.to_frame()
.assign(null_count=df.isnull().sum())
.sort_values('null_rate', ascending=False)
)
print(null_summary[null_summary['null_rate'] > 0].to_string())Section 3: univariate distributions
Every numeric column gets a histogram and a percentile summary. Every categorical column gets a value count with frequency. The goal is to find the unexpected: a numeric column with values that are all identical (a constant), a categorical column with a value that represents 99% of the data (near-constant), a distribution that has two peaks (bimodal, possibly two different populations mixed together).
numeric_cols = df.select_dtypes(include=[np.number]).columns.tolist()
# Percentile profile: more informative than describe() for skewed data
pct = df[numeric_cols].quantile([.01, .05, .10, .25, .50, .75, .90, .95, .99])
print(pct.T.to_string())
# Skewness: flag columns where mean and median diverge significantly
skew = df[numeric_cols].skew().sort_values(ascending=False)
print("\nHighly skewed columns (|skew| > 2):")
print(skew[skew.abs() > 2])High skewness is a flag, not a problem. It means the mean is not a representative summary for that column, and any downstream analysis using mean-based methods (t-test, linear regression) should either transform the column (log transform for revenue, latency, and similar right-skewed variables) or use non-parametric alternatives.
Section 4: duplicates and key integrity
Before any join or aggregation, I verify that the columns that should be unique actually are. A fact table with duplicate primary keys silently inflates every aggregate built on it.
# Primary key uniqueness
pk = 'order_id'
dup_count = df.duplicated(subset=[pk]).sum()
print(f"Duplicate {pk}: {dup_count} ({dup_count/len(df):.2%})")
# Composite key check for grain verification
grain = ['order_id', 'product_id'] # adjust per dataset
dup_grain = df.duplicated(subset=grain).sum()
print(f"Duplicate {grain}: {dup_grain}")Section 5: temporal patterns
For any dataset with a date or timestamp column, I plot the volume over time before doing any other analysis. A dataset described as "complete from 2022 to 2024" that has a suspicious gap in mid-2023 is an incomplete dataset, and any trend analysis that spans that gap is unreliable.
df['order_date'] = pd.to_datetime(df['order_purchase_timestamp'])
df['order_month'] = df['order_date'].dt.to_period('M')
monthly_counts = df.groupby('order_month').size()
monthly_counts.plot(kind='line', figsize=(12, 4), title='Orders per month')
plt.tight_layout()
# Gap detection: months with counts far below median
median_vol = monthly_counts.median()
gaps = monthly_counts[monthly_counts < median_vol * 0.5]
if not gaps.empty:
print(f"Potential data gaps: {gaps.index.tolist()}")Section 6: bivariate relationships
With distributions understood, I look at relationships between pairs of variables relevant to the analysis question. A correlation matrix for numeric columns gives a first pass; scatter plots for the most interesting pairs give the detail. The discipline here is the same as in statistics before the chart: every apparent relationship is a hypothesis until tested, and correlation is not causation.
corr = df[numeric_cols].corr(method='spearman') # Spearman: robust to non-normality
# Plot: mask upper triangle to avoid duplication
mask = np.triu(np.ones_like(corr, dtype=bool))
fig, ax = plt.subplots(figsize=(10, 8))
sns.heatmap(corr, mask=mask, annot=True, fmt='.2f', center=0,
cmap='RdBu_r', ax=ax)
ax.set_title('Spearman correlation matrix')
plt.tight_layout()Section 7: automated profiling as a complement
Libraries like ydata-profiling (formerly pandas-profiling) produce a comprehensive HTML report from a single call. The report covers distributions, missing values, correlations, and duplicate detection in one document that can be shared with stakeholders who don't read notebooks.
from ydata_profiling import ProfileReport
profile = ProfileReport(df, title="EDA Report", explorative=True, minimal=False)
profile.to_file("eda_report.html")What the template does not decide
The template runs the same way on every dataset. What it does not do is decide what the analysis means. The anomalies it surfaces, the distributions it reveals, the gaps it finds: those are inputs to a judgment that requires domain knowledge, the specific business question, and the analytical experience to know which patterns deserve investigation and which are expected features of the data.
A template is a guarantee that the exploration was systematic. It is not a guarantee that the conclusions were right. Systematic exploration eliminates the errors that come from never looking at the distribution or never checking for duplicates. It does not eliminate the errors that come from misinterpreting what the distribution means, and that is the part that cannot be templated.