US Critical Care Lakehouse on Databricks

Context

MIMIC-IV (single-hospital ICU and hospital-wide data) and eICU (200+ hospitals, multicenter) are the two most widely used credentialed critical-care research datasets in the US. Most pipelines that touch either one stop at "load it into Bronze/Silver/Gold." This project's actual claim is narrower and harder: a canonical clinical data model that lets two structurally different hospital data sources answer the same operational and clinical questions without a Gold product ever needing to know which source system a row came from.

Business Problem

Can a governed multicenter clinical lakehouse produce reproducible ICU utilization, patient deterioration, and readmission analytics across heterogeneous US hospital datasets? Answering that requires quality gates that quarantine bad data instead of dropping or passing it silently, a semantic layer that survives two sources disagreeing about what a "vital sign" table looks like, and lineage from a raw file to a Gold row.

Architecture

Four layers, not three, deliberately: a straight Bronze → Gold jump would force every Gold product to re-implement MIMIC-vs-eICU harmonization on its own. The canonical layer pays that cost once, with deterministic surrogate keys and full source traceability.

Data

• Target sources: MIMIC-IV v3.1 and the eICU Collaborative Research Database — both require individual PhysioNet credentialing (CITI training plus a signed Data Use Agreement, one per dataset). Neither has been requested yet.
• What's actually in the repo: 7 MIMIC-IV-shaped synthetic tables (111 rows total), including deliberately-bad rows exercising every quality rule, plus a larger seeded-generator variant.
• Real data from either source can never appear in this repository, in any form, even after access exists — a permanent Data Use Agreement constraint, not a temporary gap.

Methodology

1. Bronze: every source row preserved unmodified, including the bad ones, plus lineage metadata (source file, batch ID, row hash).
2. Silver: a generic quality-rule engine (structural, temporal, referential, plausibility) routes failing rows to typed quarantine tables instead of dropping them; unit standardization (e.g. Fahrenheit → Celsius) happens here.
3. Canonical: deterministic surrogate keys decouple every downstream consumer from MIMIC's own ID scheme, while source system/patient/encounter IDs stay traceable.
4. Gold: named data products with a documented grain, not ad-hoc aggregates.

Design Decisions

• Local proof uses open-source PySpark + Delta Lake, not mocks. The same dev/prod portability pattern this portfolio's AE-01 proved for DuckDB → Snowflake, applied here to local Spark/Delta → Databricks Unity Catalog.
• Why not OMOP CDM from day one: a bespoke canonical model, built and reasoned about explicitly, demonstrates canonical-modeling judgment better than adopting an industry-standard vocabulary uncritically; OMOP is a credible later extension once the foundation is proven.
• databricks bundle validate is deliberately excluded from CI: unlike terraform validate, it authenticates against a live workspace even to check syntax, so it only lives in the manually gated deploy workflows.

Results

What's verified without Databricks or real patient data:

• Bronze — 7 MIMIC tables, 111 rows, zero rows altered.
• Silver — quality-rule engine and quarantine, reconciliation proven from real quarantine-table counts: 12 admissions = 11 valid + 1 quarantined; 40 labevents = 37 valid + 3 quarantined.
• Canonical — 5 harmonized entities, schema-conformant, deterministic keys.
• Gold — icu_utilization_daily, every one of the 8 canonical ICU stays accounted for exactly once; pipeline proven idempotent across two independent runs.
• Full suite: 94/94 tests passing, evidence in docs/evidence/pytest_results.xml.
• Unity Catalog DDL, Lakeflow pipelines/jobs, and Bundle config: written and hand-reviewed, not executed — no Databricks workspace in this environment.

Tech Stack