Context
Built during Workshop 01 of Jornada de Dados (the same bootcamp series behind AE-05), this project predates the portfolio workspace. It was brought in, renamed to the portfolio's ID convention, and re-verified end-to-end on new hardware rather than just carrying over its original claims.
Business Problem
Organizations often consolidate operational data from spreadsheets and legacy systems by hand — duplicated effort, inconsistent definitions, and no real traceability from source file to published number. This pipeline demonstrates the small, repeatable alternative: ingest every workbook, apply explicit and tested transformations, and publish analysis-ready Parquet and Excel artifacts.
Architecture
Each stage is a pure function over a DataFrame in its own module (extract.py, transform.py, load.py). The CLI runner (app/main.py) only orchestrates and logs — it never processes data itself — running the matching pytest module after every stage, then a final end-to-end integration test, and writing a timestamped log. Workbook discovery and Parquet loading both sort file paths explicitly, so two runs on the same input never silently diverge in row order, and output writers propagate exceptions instead of reporting a false success.
Challenges
Why this isn't DE-01
The portfolio already reserves a slot for an ETL project: DE-01, specifically an API-sourced, cloud-warehouse, Airflow-orchestrated pipeline — and the only track item flagged as the portfolio's biggest orchestration gap. This project has none of that: a local Excel source, no cloud target, no scheduler. Folding it into DE-01 would have papered over the gap instead of closing it, so it became its own entry (DE-08) instead — a scope call worth stating plainly rather than stretching a spec to fit whatever code is on hand.
Proving it, not just running it
A polished README is not evidence. Getting this project portfolio-ready meant a clean clone, a from-scratch environment, and an actual end-to-end run — not a re-statement of the original repo's claims. That run is what produced the row counts below and the committed evidence in docs/evidence/.
Results
From a clean clone: uv venv --python 3.11 → pip install -r requirements.txt → pytest → python -m app.main.
| Stage | Result |
|---|---|
| Tests | 4/4 passed (extract, transform, load, full-pipeline integration) |
| Extract | 50 workbooks → 500 rows, 7 columns, 0 null rows |
| Transform | 500 → 500 rows (nothing to drop), expands to 9 columns (splits the ausence timestamp, formats currency, normalizes numerics) |
| Load | Parquet and Excel outputs both round-trip to the identical (500, 9) shape |
Full run log and pytest output committed at docs/evidence/ in the repo — every number above traces back to that artifact, not to a re-typed summary.
Tech Stack
| Category | Tool |
|---|---|
| Language | Python 3.11 |
| Data handling | pandas, PyArrow (Parquet), openpyxl (Excel) |
| Testing | pytest, 4 modules (unit + integration) |
| Packaging | Poetry (locked), pip fallback via requirements.txt |