Notes from building
The decisions behind the projects, the methods that make data trustworthy, and what the market says about the work.
Filter articles by tag
Showing all 36 articles.
All articles
Lessons from Building a Modern Data Stack End-to-End
Grain, idempotency, warehouse portability, and why PASS=29 is not a test count.
Delta Lake Internals: What Actually Happens When You Write, Update, and Delete
What the _delta_log is, why Parquet files are never modified in place, and what makes Time Travel and ACID guarantees possible.
Data Governance: The Eight Pillars Behind Trusted, Actionable Data
Most organizations don't have a data problem. They have a data trust problem. The framework that closes the gap between having data and being able to act on it.
What the Data Job Market Actually Asks For
Reading 262,000 real job postings instead of opinions, and where to start.
The Analytics Engineer's Toolkit
What each tool in the modern data stack is for, and when you don't need it.
How an Analytics Engineer Works: Method Over Tools
From the business question to a green build: the method that survives a tool migration.
Bad Requirements Cost More Than You Think: How I Structure Elicitation
Techniques, a validation checklist, and the mistakes that show up in UAT instead of in planning.
BPMN in Practice: Mapping Processes Before Automating Anything
Why modeling the as-is first is discipline, not bureaucracy, and what you find that prevents automating the wrong thing.
KPIs That Survive the Boardroom: Validating Metrics Against Bias and Vanity
The five tests any proposed metric must pass before it earns a place on a dashboard.
Testable User Stories: Acceptance Criteria as a Quality Contract
Why vague acceptance criteria are a deferred argument, and how to write the kind that maps to exactly one test case.
Gap Analysis with Evidence, Not Opinion
Frameworks, documented premises, and how to tell the difference between a large gap and the actual constraint.
A Defensible Business Case: ROI, Sensitivity, and Scenarios
Showing the reasoning, not just the number, and why a model that only works at best-case assumptions isn't a business case.
Stakeholder Management in High-Stakes Environments
What changes when stakeholders have competing mandates, different reporting lines, and the authority to stop your project.
Requirements Traceability: From Request to Deploy
The traceability matrix as an audit tool, and why it earns its place the day someone asks where a feature came from.
Data-Driven BA: Validating Business Hypotheses with SQL
Why querying the evidence before writing the requirement changes what you write, with three validation patterns.
Project Post-Mortem: What I Do When Delivery Misses the Mark
A structured process for learning when delivery doesn't match expectations, and the discipline to act on the findings.
Analysis Workflow: From Business Question to Actionable Insight
The six-stage process that keeps exploratory work from turning into a notebook no one can reproduce or explain.
SQL as Code: Formatting, Reviews, and Sanity Checks That Catch Semantic Errors
Why SQL style matters less than SQL discipline, and the four sanity checks that catch the errors no linter finds.
Statistics Before the Chart: Distribution, Outliers, and the Right Test
Four checks that run before any visualization, and why choosing a test after seeing the data changes what the result means.
Honest Data Visualization: Chart Choices That Preserve the Truth
The chart choices that preserve what the data actually says, and the distortions that trade accuracy for persuasion.
Systematic EDA with Python: My Reproducible Exploration Template
A structured template with pandas and profiling: the sections that run on every new dataset, in order, and why each earns its place.
Correlation Is Not Causation: Oil, Bitcoin, ADF, Granger, and Statistical Humility
How a 0.71 correlation collapsed to 0.09 once stationarity was tested, and the correct sequence of tests for time series relationships.
Well-Designed A/B Tests: Statistical Power, MDE, and When Not to Trust the Result
Why most A/B tests are decided before the data is collected, and the conditions under which a significant result is still wrong.
Dashboards with Data Contracts: What I Validate Before Publishing
The freshness, completeness, and consistency checks that run before any dashboard goes live, and why a dashboard without a contract is a liability.
Time Series with Correct Temporal Validation: Why the Common Train/Test Split Fails
How temporal leakage produces models that look good on paper and fail in production, and the validation patterns that actually hold.
Documenting Analyses for Others: Notebooks That Explain Themselves
Reproducibility as professional respect: the conventions and structural choices that make an analysis legible to whoever opens it next.
Pipelines I Trust Blindly: Idempotency, Retries, and Defensive Engineering
The three properties every production pipeline must have: idempotency that makes re-runs safe, retries that distinguish transient from structural, and alerts that fire before a stakeholder does.
Consuming Real APIs with Resilience: Rate Limits, Pagination, and Fallbacks
Patterns from real integrations with CoinGecko, Open-Meteo, and OpenSky for handling every failure mode that actually occurs in production.
Batch vs. Streaming: How I Decide, with Trade-offs Documented
The latency question that determines the architecture, what streaming actually costs, why micro-batch solves most real requirements, and the ADR that records the decision.
Airflow in Production: Testable DAGs, Sensors, and SLAs
How to write DAGs that can be tested without running them, sensors that wait correctly, and SLA miss callbacks that change incident response.
Testing Data Pipelines: Unit, Integration, Schema Validation, and Contracts
The testing pyramid for data: unit tests for transformations, integration tests on representative samples, schema contracts, and Great Expectations for continuous quality.
Data Lake, Warehouse, and Lakehouse: Architecture Chosen with Criteria, Not Hype
What each storage architecture actually provides, the decision criteria that determine which fits which workload, and why the newest option is not always the right one.
Parquet and Columnar Formats: Partitioning and Optimizations I Measured
Column pruning, predicate pushdown, partitioning strategy, the small file problem, and benchmarks comparing Snappy versus Zstd versus uncompressed on real workloads.
Data Observability: Monitoring Freshness, Volume, and Distribution Before the User Does
The three monitoring layers that catch data quality issues before stakeholders do, and the alerting architecture that makes each finding immediately actionable.
Infrastructure as Code: Docker and Total Reproducibility of My Projects
The clone-build-run discipline: Dockerfile layer ordering, docker-compose for the full local stack, and the .env.example pattern that handles secrets without a manual step.
FinOps for Data Engineering: Estimating and Controlling Costs in AWS
Storage tier lifecycle policies, scan-based query cost modeling, compute right-sizing with spot instances, and tagging for attribution before the bill surprises anyone.