https://andrematiello.com/blog/ Articles about analytics engineering, data engineering, analytics, and business analysis. en-us Tue, 28 Jul 2026 12:00:00 -0300 https://andrematiello.com/blog/pipelines-idempotency-retries-alerts.html https://andrematiello.com/blog/pipelines-idempotency-retries-alerts.html The three properties every production pipeline must have before I stop watching it: idempotency that makes re-runs safe, retries that distinguish transient from structural failures, and alerts that fire before a stakeholder does. Data Engineering matiello.andre@hotmail.com (André Matiello Caramanti) Mon, 20 Jul 2026 20:26:01 +0000 https://andrematiello.com/blog/consuming-apis-with-resilience.html https://andrematiello.com/blog/consuming-apis-with-resilience.html The patterns I use with CoinGecko, Open-Meteo, and OpenSky to handle rate limits gracefully, page through large result sets without data loss, and keep pipelines running when the upstream source does not. Data Engineering matiello.andre@hotmail.com (André Matiello Caramanti) Mon, 20 Jul 2026 20:26:01 +0000 https://andrematiello.com/blog/batch-vs-streaming.html https://andrematiello.com/blog/batch-vs-streaming.html The decision framework I use to choose between batch, micro-batch, and streaming, why the architecture decision belongs in a versioned document, and the trade-offs that are real versus the ones that exist only in conference talks. Data Engineering matiello.andre@hotmail.com (André Matiello Caramanti) Mon, 20 Jul 2026 20:26:01 +0000 https://andrematiello.com/blog/airflow-in-production.html https://andrematiello.com/blog/airflow-in-production.html What I learned from orchestrating real data loads with Airflow: how to write DAGs that can be tested without running them, how sensors prevent upstream dependency failures from silently propagating, and how SLA miss callbacks change incident response. Data Engineering matiello.andre@hotmail.com (André Matiello Caramanti) Mon, 20 Jul 2026 20:26:01 +0000 https://andrematiello.com/blog/testing-data-pipelines.html https://andrematiello.com/blog/testing-data-pipelines.html The testing pyramid for data pipelines: unit tests for transformation logic, integration tests on representative samples, schema contracts that catch upstream changes before they reach production, and Great Expectations for continuous data quality. Data Engineering matiello.andre@hotmail.com (André Matiello Caramanti) Mon, 20 Jul 2026 20:26:01 +0000 https://andrematiello.com/blog/data-lake-warehouse-lakehouse.html https://andrematiello.com/blog/data-lake-warehouse-lakehouse.html What each storage architecture actually provides and costs, the decision criteria that determine which one fits which workload, and why the Lakehouse is not always the answer just because it is the newest. Data Engineering matiello.andre@hotmail.com (André Matiello Caramanti) Mon, 20 Jul 2026 20:26:01 +0000 https://andrematiello.com/blog/parquet-columnar-formats.html https://andrematiello.com/blog/parquet-columnar-formats.html Why Parquet outperforms CSV for analytical workloads, the partitioning strategies that make or break query performance, the small file problem and how to fix it, and the benchmarks I ran to verify what the documentation claims. Data Engineering matiello.andre@hotmail.com (André Matiello Caramanti) Mon, 20 Jul 2026 20:26:01 +0000 https://andrematiello.com/blog/data-observability.html https://andrematiello.com/blog/data-observability.html The monitoring layer that catches data quality issues before stakeholders do: freshness checks with SLA thresholds, volume anomaly detection with rolling baselines, distribution monitoring for schema drift and null rate changes, and the alerting architecture that makes it actionable. Data Engineering matiello.andre@hotmail.com (André Matiello Caramanti) Mon, 20 Jul 2026 20:26:01 +0000 https://andrematiello.com/blog/infrastructure-as-code-docker.html https://andrematiello.com/blog/infrastructure-as-code-docker.html Why every data project I build ships with a Dockerfile and docker-compose: the clone-build-run discipline that eliminates environment drift, makes onboarding a one-command operation, and treats reproducibility as a first-class deliverable. Data Engineering matiello.andre@hotmail.com (André Matiello Caramanti) Mon, 20 Jul 2026 20:26:01 +0000 https://andrematiello.com/blog/finops-data-engineering.html https://andrematiello.com/blog/finops-data-engineering.html How I think about cloud cost before writing the first line of a pipeline: storage tier decisions, compute right-sizing, query cost modeling for scan-based pricing, tagging for attribution, and the discipline of treating cost as a first-class engineering concern. Data Engineering matiello.andre@hotmail.com (André Matiello Caramanti) Mon, 20 Jul 2026 20:26:01 +0000 https://andrematiello.com/blog/analysis-workflow-question-to-insight.html https://andrematiello.com/blog/analysis-workflow-question-to-insight.html A documented, repeatable workflow for turning a business question into a defensible insight: the stages, the checkpoints where I stop and verify, and the mistakes each one prevents. Method matiello.andre@hotmail.com (André Matiello Caramanti) Mon, 20 Jul 2026 20:16:27 +0000 https://andrematiello.com/blog/sql-as-code-peer-review.html https://andrematiello.com/blog/sql-as-code-peer-review.html Why SQL written for analysis deserves the same discipline as production code: formatting standards, sanity tests, peer review, and the mistakes that only a second reader catches. Analytics matiello.andre@hotmail.com (André Matiello Caramanti) Mon, 20 Jul 2026 20:16:27 +0000 https://andrematiello.com/blog/statistics-before-the-chart.html https://andrematiello.com/blog/statistics-before-the-chart.html The statistical checks that run before any visual goes into a report: normality, outliers, and significance testing done in the right order, with the right humility. Statistics matiello.andre@hotmail.com (André Matiello Caramanti) Mon, 20 Jul 2026 20:16:27 +0000 https://andrematiello.com/blog/honest-data-visualization.html https://andrematiello.com/blog/honest-data-visualization.html The chart choices that preserve what the data actually says, the distortions I refuse to make, and the principle that a misleading chart produced with clean data is still a lie. Visualization matiello.andre@hotmail.com (André Matiello Caramanti) Mon, 20 Jul 2026 20:16:27 +0000 https://andrematiello.com/blog/systematic-eda-with-python.html https://andrematiello.com/blog/systematic-eda-with-python.html A structured, reproducible exploratory data analysis template with pandas and profiling: the sections that run on every new dataset, in order, and why each one earns its place. Analytics matiello.andre@hotmail.com (André Matiello Caramanti) Mon, 20 Jul 2026 20:16:27 +0000 https://andrematiello.com/blog/correlation-not-causation.html https://andrematiello.com/blog/correlation-not-causation.html How I tested a spurious correlation between oil prices and Bitcoin, what the ADF test and Granger causality actually check, and why the hardest part is accepting what the data doesn't say. Statistics matiello.andre@hotmail.com (André Matiello Caramanti) Mon, 20 Jul 2026 20:16:27 +0000 https://andrematiello.com/blog/well-designed-ab-tests.html https://andrematiello.com/blog/well-designed-ab-tests.html Why most A/B tests are decided before the data is collected, what statistical power and minimum detectable effect mean in practice, and the conditions under which a significant result is still wrong. Statistics matiello.andre@hotmail.com (André Matiello Caramanti) Mon, 20 Jul 2026 20:16:27 +0000 https://andrematiello.com/blog/dashboards-with-data-contracts.html https://andrematiello.com/blog/dashboards-with-data-contracts.html The freshness, completeness, and consistency checks that run before any dashboard goes live, and why a dashboard without a data contract is a liability dressed as an asset. Analytics matiello.andre@hotmail.com (André Matiello Caramanti) Mon, 20 Jul 2026 20:16:27 +0000 https://andrematiello.com/blog/time-series-temporal-validation.html https://andrematiello.com/blog/time-series-temporal-validation.html Why the standard random train/test split breaks on time series data, how temporal leakage produces models that look good on paper and fail in production, and the validation patterns that actually hold. Statistics matiello.andre@hotmail.com (André Matiello Caramanti) Mon, 20 Jul 2026 20:16:27 +0000 https://andrematiello.com/blog/documenting-analyses-for-others.html https://andrematiello.com/blog/documenting-analyses-for-others.html Reproducibility as professional respect: the notebook conventions, documentation habits, and structural choices that make an analysis legible to the person who opens it six months later. Analytics matiello.andre@hotmail.com (André Matiello Caramanti) Mon, 20 Jul 2026 20:16:27 +0000 https://andrematiello.com/blog/data-governance-eight-pillars.html https://andrematiello.com/blog/data-governance-eight-pillars.html Most organizations don't have a data problem. They have a data trust problem. Here is the framework that closes the gap between having data and being able to act on it. Data Engineering matiello.andre@hotmail.com (André Matiello Caramanti) Sun, 19 Jul 2026 23:57:17 +0000 https://andrematiello.com/blog/delta-lake-internals.html https://andrematiello.com/blog/delta-lake-internals.html Most engineers use Delta tables daily without knowing what the _delta_log is, how updates work on immutable Parquet files, or what makes Time Travel possible. Here's what's underneath. Data Engineering matiello.andre@hotmail.com (André Matiello Caramanti) Sun, 19 Jul 2026 23:52:20 +0000 https://andrematiello.com/blog/bad-requirements-cost-more-than-you-think.html https://andrematiello.com/blog/bad-requirements-cost-more-than-you-think.html Techniques, a validation checklist, and the mistakes I learned to stop making before the first line of specification is written. Requirements matiello.andre@hotmail.com (André Matiello Caramanti) Sun, 19 Jul 2026 09:17:12 +0000 https://andrematiello.com/blog/bpmn-in-practice.html https://andrematiello.com/blog/bpmn-in-practice.html Why modeling first is discipline, not bureaucracy, and what you find in the as-is state that saves you from automating the wrong thing. Process Analysis matiello.andre@hotmail.com (André Matiello Caramanti) Sun, 19 Jul 2026 09:17:12 +0000 https://andrematiello.com/blog/kpis-that-survive-the-boardroom.html https://andrematiello.com/blog/kpis-that-survive-the-boardroom.html The tests I run on any proposed metric before it earns a place on a dashboard, and why most metrics fail at least one of them. Measurement matiello.andre@hotmail.com (André Matiello Caramanti) Sun, 19 Jul 2026 09:17:12 +0000 https://andrematiello.com/blog/testable-user-stories.html https://andrematiello.com/blog/testable-user-stories.html The bridge between what was asked for and what gets tested, and why vague acceptance criteria are a deferred argument about who is right. Requirements matiello.andre@hotmail.com (André Matiello Caramanti) Sun, 19 Jul 2026 09:17:12 +0000 https://andrematiello.com/blog/gap-analysis-with-evidence.html https://andrematiello.com/blog/gap-analysis-with-evidence.html How to move from 'we have a problem' to a structured current-state and future-state comparison that survives a room full of stakeholders. Analysis matiello.andre@hotmail.com (André Matiello Caramanti) Sun, 19 Jul 2026 09:17:12 +0000 https://andrematiello.com/blog/defensible-business-case.html https://andrematiello.com/blog/defensible-business-case.html Showing the reasoning, not just the number, and why a model that only works at best-case assumptions is not a business case. Business Analysis matiello.andre@hotmail.com (André Matiello Caramanti) Sun, 19 Jul 2026 09:17:12 +0000 https://andrematiello.com/blog/stakeholder-management-high-stakes.html https://andrematiello.com/blog/stakeholder-management-high-stakes.html What changes when stakeholders have competing mandates, different reporting lines, and the authority to stop your project unilaterally. Leadership matiello.andre@hotmail.com (André Matiello Caramanti) Sun, 19 Jul 2026 09:17:12 +0000 https://andrematiello.com/blog/requirements-traceability.html https://andrematiello.com/blog/requirements-traceability.html The traceability matrix as an audit tool, and why it earns its place the day someone asks where that feature came from. Requirements matiello.andre@hotmail.com (André Matiello Caramanti) Sun, 19 Jul 2026 09:17:12 +0000 https://andrematiello.com/blog/data-driven-ba.html https://andrematiello.com/blog/data-driven-ba.html The bridge between the BA role and the data stack, and why querying the evidence before writing the requirement changes what you write. Analytics matiello.andre@hotmail.com (André Matiello Caramanti) Sun, 19 Jul 2026 09:17:12 +0000 https://andrematiello.com/blog/project-post-mortem.html https://andrematiello.com/blog/project-post-mortem.html A culture of improvement starts with a structured process for learning when a delivery doesn't match what was expected, and with the discipline to act on what you find. Retrospective matiello.andre@hotmail.com (André Matiello Caramanti) Sun, 19 Jul 2026 09:17:12 +0000 https://andrematiello.com/blog/lessons-from-building-a-modern-data-stack.html https://andrematiello.com/blog/lessons-from-building-a-modern-data-stack.html What a dlt → DuckDB/Snowflake → dbt pipeline taught me about grain, idempotent ingestion, warehouse portability, and reading your own build output honestly. Analytics Engineering matiello.andre@hotmail.com (André Matiello Caramanti) Wed, 15 Jul 2026 00:24:49 -0300 https://andrematiello.com/blog/what-the-data-job-market-actually-asks-for.html https://andrematiello.com/blog/what-the-data-job-market-actually-asks-for.html A read of 83,517 data analyst and 178,738 data engineer postings: which skills employers name, why the number of skills per posting matters more than the list, and where that says to start. Career matiello.andre@hotmail.com (André Matiello Caramanti) Wed, 15 Jul 2026 00:24:49 -0300 https://andrematiello.com/blog/the-analytics-engineers-toolkit.html https://andrematiello.com/blog/the-analytics-engineers-toolkit.html What each tool in the modern data stack is actually for: SQL, dbt, ingestion, warehouses, tests, orchestration, semantic layers, and when you don't need it yet. Tools matiello.andre@hotmail.com (André Matiello Caramanti) Wed, 15 Jul 2026 00:24:49 -0300 https://andrematiello.com/blog/how-an-analytics-engineer-works.html https://andrematiello.com/blog/how-an-analytics-engineer-works.html The working method behind a pipeline people trust: the business question first, the grain before the SQL, deliberate layering, tests aimed at what breaks silently, and evidence instead of adjectives. Method matiello.andre@hotmail.com (André Matiello Caramanti) Wed, 15 Jul 2026 00:24:49 -0300