Data Lake, Warehouse, and Lakehouse: Architecture Chosen with Criteria, Not Hype

The architecture choice between a data lake, a data warehouse, and a lakehouse is not a question of which is better in general. It is a question of which properties the workload actually requires and which constraints the team must operate within. The wrong choice is not the one that picks the older technology. It is the one that makes the choice based on anything other than a documented analysis of requirements.

What a data lake actually provides

A data lake is object storage (S3, GCS, Azure Blob) with files in any format, organized by a naming convention rather than a schema enforced by the storage layer. The defining properties are low storage cost, schema-on-read, and the ability to store anything: structured, semi-structured, and unstructured data. Data is written as files. Consumers define the schema at query time.

The advantages are real: storing a petabyte of raw log data in Parquet on S3 costs a fraction of what the same data would cost in a cloud warehouse. The raw data is preserved in its original form, which matters for ML training (where the model may benefit from signals the current pipeline discards) and for compliance (where the original record may be needed for audit).

The disadvantages are equally real and more often understated. Schema-on-read means no enforcement: a producer that changes a column name silently breaks every consumer that read that column, with no error at write time. ACID transactions are not native to object storage: a write that fails halfway through leaves partial files with no automatic rollback. Concurrent writes without coordination produce undefined state. Query performance on object storage depends heavily on file layout, format, and partition design, which are the responsibility of the producer.

What a data warehouse provides

A cloud data warehouse (BigQuery, Redshift, Snowflake) is a managed OLAP system with schema enforcement, SQL as the primary interface, and storage and compute designed for analytical query patterns. Data is written in structured form after transformation. The warehouse enforces types, constraints, and schema at write time.

-- Warehouse: schema enforced at load time
CREATE TABLE fct_orders (
    order_id     VARCHAR(36) NOT NULL,
    order_date   DATE        NOT NULL,
    revenue_usd  DECIMAL(12,2),
    status       VARCHAR(20) CHECK (status IN ('completed','cancelled','pending')),
    PRIMARY KEY (order_id)
);

-- A load that violates the schema fails at INSERT, not at query time
INSERT INTO fct_orders VALUES ('abc', '2024-08-01', -50.00, 'completed');
-- ERROR: CHECK constraint violation on revenue_usd (if constraint were added)

The warehouse is the right answer for teams whose primary consumers are SQL users running analytical queries, where data correctness is a hard requirement and not negotiable, and where storage volumes are in the terabyte range rather than petabyte. The cost per query in a warehouse is higher than in a lake, and the storage cost per byte is higher, but these are the right trade-offs when the query frequency is high and the correctness requirement is strict.

What the Lakehouse adds

The Lakehouse is an architectural pattern (implemented by Delta Lake, Apache Iceberg, and Apache Hudi) that adds ACID transaction semantics, schema enforcement, and versioning on top of object storage. The storage layer remains cheap (Parquet files on S3), but a transaction log coordinates writes and provides the guarantees that raw object storage lacks.

# Delta Lake: ACID write on object storage
from delta.tables import DeltaTable
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
    .getOrCreate()

# Upsert (MERGE): atomic, ACID, safe to re-run
delta_table = DeltaTable.forPath(spark, "s3://data-lake/gold/fct_orders")
new_data = spark.createDataFrame(load_new_orders())

delta_table.alias("target").merge(
    new_data.alias("source"),
    "target.order_id = source.order_id"
).whenMatchedUpdateAll() \
 .whenNotMatchedInsertAll() \
 .execute()

# Time travel: read the table as it was 3 versions ago
df_historical = spark.read.format("delta") \
    .option("versionAsOf", delta_table.history(3).first()["version"]) \
    .load("s3://data-lake/gold/fct_orders")

The Lakehouse is the right answer when the workload mixes ML (which benefits from the raw data access and low storage cost of a lake) with analytical reporting (which benefits from the ACID guarantees and schema enforcement of a warehouse), and when the data volume is large enough that warehouse storage cost is a real constraint.

Decision criteria

The hybrid reality

Most mature data platforms use all three layers at once, with a deliberate boundary between them. Raw data lands in the lake (cheap, no transformation, exact replica of the source). Curated, transformed data lives in the warehouse or lakehouse (enforced schema, ACID, optimized for queries). The distinction is not about which system is better: it is about which layer is responsible for which type of data at which stage of the pipeline.

The architecture choice is a constraint on everything built on top of it. It is worth the time to document the decision with explicit criteria, alternatives considered, and trade-offs accepted, before the first pipeline is built against the chosen storage layer.