Batch vs. Streaming: How I Decide, with Trade-offs Documented

The batch versus streaming debate is one of the most cargo-culted in data engineering. Teams adopt streaming because it sounds modern, discover that exactly-once semantics are hard, stateful processing is expensive to operate, and their dashboards refresh daily anyway. Other teams stay in batch long after their latency requirements have outgrown it, and nobody notices because nobody documented what the latency requirement actually was.

The decision is not primarily technical. It is driven by one question with a quantitative answer: what is the maximum acceptable latency between an event occurring and the data about that event being available to consumers? The answer to that question determines the architecture. The architecture does not determine the answer.

The latency question and what it implies

Latency requirements fall into three practical ranges, each with a corresponding architecture:

Most enterprise analytics use cases fall into the third category. A daily sales report that refreshes at 6 a.m. has a latency requirement of 24 hours. Building a streaming pipeline to serve it is adding complexity for no benefit to the user. The user does not need the data sooner: they need it reliably, correctly, and available when they open their dashboard in the morning.

What streaming actually costs

Streaming is genuinely the right choice for latency requirements under one hour, but the costs are real and often understated in architectural discussions.

Exactly-once semantics (guaranteeing that each event is processed exactly once, not at-least-once or at-most-once) requires coordination between the message broker, the processing framework, and the destination. In Kafka plus Flink, this is achievable but requires careful configuration of transactions, offsets, and sink idempotency. A misconfigured streaming pipeline is harder to diagnose than a misconfigured batch job because the state is distributed and the failure modes are not always reproducible.

# Spark Structured Streaming: micro-batch with a trigger interval
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, window, sum as _sum

spark = SparkSession.builder.appName("order_aggregation").getOrCreate()

orders_stream = (
    spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "broker:9092")
    .option("subscribe", "raw_orders")
    .load()
)

# Parse JSON, aggregate in 5-minute tumbling windows
aggregated = (
    orders_stream
    .selectExpr("CAST(value AS STRING) AS json")
    .select(col("json").cast("struct").alias("data"))
    .select("data.*")
    .withWatermark("ts", "10 minutes")  # tolerate late arrivals up to 10 min
    .groupBy(window("ts", "5 minutes"))
    .agg(_sum("amount").alias("total_revenue"))
)

# Trigger: process every 5 minutes (micro-batch, not continuous)
query = (
    aggregated
    .writeStream
    .outputMode("append")
    .format("delta")
    .option("checkpointLocation", "/checkpoints/order_agg")
    .trigger(processingTime="5 minutes")
    .start()
)

The watermark is the mechanism that handles late-arriving events: events that arrive up to 10 minutes after their event timestamp are included in the correct window. Events that arrive later are dropped. The trade-off is explicit: accepting late data up to a threshold versus the complexity of holding windows open indefinitely. The threshold is a business decision: how late can an order event plausibly arrive, and does including it in the aggregate change any decision the consumer will make?

Micro-batch as the practical middle ground

Micro-batch (a streaming job triggered every N minutes rather than processing events continuously) has most of the latency benefits of streaming for use cases that need sub-hourly freshness, with significantly simpler failure modes. A micro-batch job that fails is a batch job that failed: the state is a checkpoint file, the re-run replays from the last committed offset, and the debugging toolset is the same as for any other scheduled job.

For most near-real-time requirements I have encountered in practice, the question "do you need data in 30 seconds or 5 minutes?" is answered with "we thought 30 seconds, but actually 5 minutes is fine for what we're doing." That 5-minute latency is achievable with a triggered micro-batch job at a fraction of the operational complexity of a true streaming system.

The architecture decision as a versioned artifact

Architecture Decision Records (ADRs) are short documents that capture why a significant technical choice was made. They are not design documents or technical specifications. They are decision logs: what was decided, what alternatives were considered, what trade-offs were accepted, and who was in the room.

# ADR-003: Order Aggregation Pipeline Architecture

## Status: Accepted (2024-09-12)

## Context
The product team needs order revenue aggregated and available for the
operational dashboard with a latency of no more than 15 minutes. Current
batch pipeline refreshes hourly.

## Decision
Use Spark Structured Streaming in micro-batch mode, triggered every 5 minutes.

## Alternatives considered
1. **True streaming (Flink with Kafka)**: achieves ~5s latency but requires
   Flink cluster management, exactly-once configuration, and monitoring
   expertise we do not currently have. Latency requirement is 15 minutes,
   so 5s latency provides no benefit over 5 minutes.
2. **Reduce batch interval to 15 minutes**: simpler, but re-processes the
   full date partition every 15 minutes, increasing warehouse compute cost
   by ~4x per day.

## Trade-offs accepted
- Micro-batch does not handle late arrivals as precisely as true streaming.
  We accept events up to 10 minutes late; later arrivals are excluded.
- Checkpoint management adds operational overhead not present in batch.

## Consequences
- Latency target of 15 minutes met by the 5-minute trigger interval.
- Monitoring required for checkpoint lag and consumer group offset.
- Review in 90 days: if latency requirement tightens below 5 minutes,
  re-evaluate true streaming.

The ADR does not need to be long. It needs to capture the decision in enough detail that an engineer who joins the team in 18 months can read it and understand why the architecture is the way it is, what was considered, and what would trigger a reconsideration. Without this document, the knowledge is oral, the rationale is lost when people leave, and every new engineer reopens the same decision from scratch.