Delta Lake Internals: What Actually Happens When You Write, Update, and Delete

Delta Lake is one of those technologies that works well enough that most people never have to understand it. You write a DataFrame, Spark produces a table, queries return the right rows, and the pipeline runs every morning without incident. The internals stay invisible, which is comfortable until something goes wrong: a pipeline that failed halfway through and left partial data, a version query that returns an unexpected row count, a VACUUM that deleted files you needed. At that point, knowing what Delta is actually doing makes the difference between a twenty-minute fix and a two-hour investigation.

The single most important thing to understand about Delta Lake is stated cleanly in the documentation and ignored almost universally in practice: Delta is not a file format. It is a storage layer built on top of Parquet. Understanding that one distinction resolves most of the confusion about how it works.

What Delta Lake actually is

A Delta table is two things stored together in the same directory:

Parquet files that contain the actual data rows.
A transaction log (the _delta_log folder) that records every operation ever performed on the table.

The Parquet files are the storage medium. The transaction log is what turns a folder of Parquet files into a reliable, ACID-compliant table with versioning, concurrent writes, and rollback capability. Remove the transaction log and you have a collection of Parquet files. Add the transaction log and you have Delta Lake.

sales_delta/
├── _delta_log/
│   ├── 00000000000000000000.json
│   ├── 00000000000000000001.json
│   ├── 00000000000000000002.json
│   └── 00000000000000000100.checkpoint.parquet
├── part-00001.parquet
├── part-00002.parquet
├── part-00003.parquet
└── part-00004.parquet

Every file in _delta_log numbered sequentially with twenty digits is a transaction: a JSON record of one commit. The .checkpoint.parquet files are periodic snapshots of the accumulated log state. The .parquet files in the root directory are the data. Nothing about the data files is special: a reader that doesn't know about Delta can open any one of them with any Parquet reader and read the rows inside. What it can't do is know which rows are current, which are obsolete, and which version of the table those rows belong to. That knowledge lives entirely in the transaction log.

What happens during INSERT

When you run an INSERT, Spark does three things, in order:

Writes the new rows into one or more new Parquet files.
Writes a new JSON entry into _delta_log recording the commit: which files were added, the schema at the time of the write, the operation type, and a timestamp.
Increments the version number. The table was at version N before the insert; it is at version N+1 after.

# version 10: part-00001.parquet  (rows 1-5000)
# INSERT 2,000 new rows
# version 11: part-00001.parquet  (rows 1-5000)
#             part-00002.parquet  (rows 5001-7000)  <-- new file
# _delta_log: 00000000000000000011.json added

No existing file is touched. No existing file is rewritten. The old Parquet files are exactly as they were. This is the design constraint that shapes everything else about how Delta works: Parquet files are immutable. Delta never modifies a data file in place. Every write produces new files.

What happens during UPDATE and DELETE

This is where the immutability constraint produces behavior that surprises people. If Parquet files can't be modified, how does an UPDATE change a row's value? How does a DELETE remove a row?

The answer is: it doesn't change the existing file at all. It marks the old file as removed and writes a new file containing the correct rows.

An UPDATE on a single row in part-00001.parquet:

Reads the entire affected Parquet file.
Applies the update in memory.
Writes the modified rows (plus all the unmodified rows from the same file) into a new Parquet file: part-00005.parquet.
Records in the transaction log: part-00001.parquet is removed, part-00005.parquet is added.

The old file is not deleted from storage immediately. It is marked as logically removed in the transaction log. It remains on disk until VACUUM cleans it up. This is not a bug; it is what makes Time Travel possible.

DELETE works identically: the affected file is rewritten without the deleted rows, the old file is marked as removed, and the transaction log records the commit. A DELETE WHERE country = 'US' on a table with five data files produces up to five new Parquet files (one per affected file), marks the originals as removed, and writes a single transaction log entry that records all of it as one atomic operation.

Every write in Delta produces new files. No data file is ever modified in place. The transaction log is the authoritative record of which files belong to the current version of the table. A file that exists in the directory but is not referenced by the current log state is invisible to readers, even though it is physically present on disk.

The transaction log: the heart of Delta Lake

The _delta_log folder is where all guarantees are grounded. Every commit is a JSON file. Every JSON file records, at minimum:

The files added by this commit (add actions), with size, statistics, and the data change flag.
The files removed by this commit (remove actions), with a deletion timestamp.
The commit metadata: operation type, user, timestamp, and the parameters of the operation (the WHERE clause of a DELETE, the SET clause of an UPDATE).

// 00000000000000000003.json (simplified)
{"commitInfo": {"operation": "DELETE", "operationParameters": {"predicate": "[\"(country = 'US')\"]"}, "timestamp": 1720000000000}}
{"remove": {"path": "part-00001.parquet", "deletionTimestamp": 1720000000000}}
{"remove": {"path": "part-00002.parquet", "deletionTimestamp": 1720000000000}}
{"add": {"path": "part-00006.parquet", "size": 184320, "dataChange": true}}
{"add": {"path": "part-00007.parquet", "size": 97280, "dataChange": true}}

To read the current version of the table, a Delta reader replays the log from the beginning (or from the last checkpoint) and builds the list of files that are currently active: all files added and not subsequently removed. That list is the table. Any file not on that list is either obsolete or belongs to a future version, and the reader ignores it.

This log-replay architecture is what delivers ACID guarantees. Atomicity: a commit either produces a complete JSON log entry or nothing, so a failed write never produces a partial table state. Isolation: concurrent writers each attempt to write their own log entry to the next sequential version slot; only one can claim a given version number, so conflicts are detected and one writer retries. Consistency and durability follow from the same structure.

Checkpoints: why the log doesn't grow forever

Replaying a transaction log that has ten thousand entries on every read is expensive. Delta solves this with checkpoints: periodic Parquet snapshots of the complete log state at a given version, written every hundred commits by default.

A checkpoint file (00000000000000000100.checkpoint.parquet) contains the full set of active files at version 100: every add action not yet superseded by a remove, consolidated into a single Parquet file that can be read in one pass. When a reader wants the current table state, it finds the latest checkpoint, reads the consolidated file list from it, and then replays only the JSON log entries since the checkpoint. A table at version 1,432 with a checkpoint at version 1,400 replays thirty-two JSON files, not 1,432.

Checkpoints are written automatically. The only reason to know they exist is to understand why removing them manually (or pointing the Delta reader at an old copy of the table without its checkpoint files) forces a full log replay from version zero, which on a high-write table can be slow enough to time out.

Time Travel

Time Travel is the capability to query any previous version of the table, not just the current one. It is possible precisely because Delta never deletes old Parquet files immediately: the files removed by an UPDATE or DELETE remain on disk until VACUUM explicitly cleans them up.

-- Query the table as it was at version 3
SELECT * FROM sales_delta VERSION AS OF 3;

-- Query the table as it was at a specific timestamp
SELECT * FROM sales_delta TIMESTAMP AS OF '2024-09-01 00:00:00';

To serve a version-3 query, the reader replays the transaction log up to and including the version-3 entry, builds the file list that was active at that point, and reads from those files. The files are still there (assuming VACUUM hasn't run with a retention window shorter than the query lookback); the log entry tells the reader which ones to use.

Time Travel is how you recover from a bad write. An accidental DELETE, a transform that produced the wrong output, a load that duplicated rows: any of these can be undone by reading the pre-operation version and writing it back as the current version. The operation is a full table write (new Parquet files, new log entry), but it requires no backup infrastructure and no restore procedure. The history was already there.

The practical limit on Time Travel is VACUUM. Once VACUUM removes the old files, the version that referenced them can no longer be fully reconstructed. The default retention is seven days. Reducing it below seven days is possible but requires understanding that every version query before the new retention window will fail with a file-not-found error on the Parquet files VACUUM removed.

OPTIMIZE and VACUUM: table maintenance

Delta tables accumulate small files over time. Every streaming micro-batch, every incremental load, every single-row insert produces at least one Parquet file. A table with ten thousand small files takes longer to query than a table with ten larger files that contain the same rows, because each file open is a storage round trip, and ten thousand of them add up. OPTIMIZE compacts the small files into larger ones.

OPTIMIZE sales_delta;

-- With Z-ordering: co-locate related rows physically on disk
OPTIMIZE sales_delta ZORDER BY (country, sale_date);

OPTIMIZE writes new, larger Parquet files and marks the small files as removed in the transaction log. It does not delete the small files from disk: that is VACUUM's job. The data is unchanged; the file layout is different. The old small files remain available for Time Travel queries until VACUUM removes them.

VACUUM permanently deletes from storage all files that are marked as removed in the transaction log and older than the retention threshold.

-- Remove files removed more than 7 days ago (default)
VACUUM sales_delta;

-- Dry run: show what would be deleted without deleting it
VACUUM sales_delta DRY RUN;

VACUUM is irreversible. Files it deletes cannot be recovered through Time Travel because Time Travel requires the physical files, not just the log entries that referenced them. The DRY RUN flag is not optional etiquette: it is the check you run before every production VACUUM to confirm that the files about to be deleted are not files any concurrent reader or pending Time Travel query needs.

The complete lifecycle

Put together, the lifecycle of a Delta table is a loop:

Write data (INSERT, UPDATE, DELETE, MERGE) creates new Parquet files and transaction log entries. The table version increments with every commit. Read data replays the log to the current version (or a past version), builds the active file list, and reads from those files. Checkpoint creation happens automatically every hundred commits, consolidating the log state for faster future reads. OPTIMIZE runs periodically to compact small files, improving query performance without changing the data. VACUUM runs periodically to remove files that are no longer needed by any supported Time Travel query, keeping storage costs from growing unboundedly.

Understanding this loop makes the behavior of every Delta Lake command predictable. MERGE is an UPDATE and a DELETE in one atomic commit: new files for matched rows, removed files for the ones replaced, a single log entry. Schema evolution adds a column to the table schema recorded in the next log entry, and all existing files are read with the new schema applied (null for the missing column). Change Data Feed writes additional log entries recording the before-and-after state of every changed row, which a downstream consumer can read as a stream of changes rather than a full table scan.

None of those features requires understanding anything beyond what this article covers. The architecture is the same: immutable Parquet files, an append-only transaction log, a reader that builds the current state from the log. The features are consequences of that architecture, not additions to it. Once you know what _delta_log is and why Parquet files are never modified in place, the rest follows.