TL;DR: Choose by required freshness vs complexity. Batch is simpler, cheaper, and easier to reason about; use it when hours-to-daily latency is fine. Streaming gives seconds-to-minutes freshness at higher operational complexity; use it when decisions are real-time (fraud, recommendations, alerting). Lambda runs both a batch and a speed layer (accurate but two codebases to maintain); Kappa runs a single streaming pipeline and reprocesses by replaying the log, avoiding the dual-code burden.
How to approach it. Anchor on the real driver: the freshness the use case actually needs, and resist defaulting to streaming because it sounds modern. Give the batch-vs-streaming tradeoff, then explain Lambda and Kappa as two answers to "how do I get both accuracy and low latency," and why the industry drifted toward Kappa.
A strong answer. Batch vs streaming. Batch processing runs on bounded chunks (hourly, nightly): simpler to build, test, and reason about, cheaper, and naturally handles reprocessing and large historical jobs; the cost is latency (results are as fresh as the last run). Streaming processes unbounded data continuously: results in seconds to minutes, but it is operationally harder (stateful processing, watermarks for late/out-of-order data, exactly-once effects, always-on infrastructure). Match the architecture to the freshness requirement: a daily revenue dashboard does not need streaming; fraud scoring, live recommendations, and alerting do. Choosing streaming when batch suffices buys complexity and cost for no benefit.
Lambda architecture. Run two layers in parallel: a batch layer that recomputes accurate views over all historical data (slow but correct and reprocessable) and a speed layer that streams recent data for low-latency, approximate results; queries merge the two. It delivers both freshness and eventual accuracy, but the well-known pain is two codebases implementing the same logic (batch and streaming) that must be kept in sync, which is a maintenance and correctness burden.
Kappa architecture. Drop the separate batch layer: treat everything as a stream over an immutable, replayable log (e.g. Kafka). For "batch" work or backfills, you reprocess by replaying the log through the same streaming code. One codebase, one mental model. It works when your stream processor can handle reprocessing at scale and you can retain/replay the log; it struggles if very large historical recomputations are cheaper in a true batch engine. Modern lakehouse tools (Spark Structured Streaming, Flink, Delta) blur the line by running the same code in batch or streaming mode, which is essentially the Kappa ideal.
The defensible position: default to batch for simplicity unless freshness demands streaming; if you need both, prefer a single-codebase (Kappa-style) approach over Lambda's dual implementation unless huge historical recomputes force a dedicated batch layer.
| Batch | Streaming | Lambda | Kappa | |
|---|---|---|---|---|
| Freshness | hours to daily | seconds to minutes | both (merged at query) | seconds to minutes |
| Codebases | one | one | two (sync burden) | one |
| Backfill | re-run the job | weak | batch layer | replay the log |
| Pick when | latency is loose | decisions are real-time | huge history recompute is far cheaper in batch | replay scales and log is retained |
Key takeaways
- Match the architecture to the freshness the use case actually needs; a daily dashboard does not earn the operational cost of an always-on stateful pipeline.
- Lambda's core flaw is two codebases (batch and speed) implementing the same logic and drifting out of sync.
- Kappa replaces the batch layer with log replay through the same streaming code: one codebase, one mental model, assuming you can retain and replay the log.
- Modern lakehouse engines (Spark Structured Streaming, Flink, Delta) run the same code in batch or streaming mode, which is the Kappa ideal in practice.
What interviewers probe next.
- "Why did Kappa emerge?" To eliminate Lambda's duplicate batch+streaming logic and the bugs from keeping two implementations consistent.
- "Exactly-once in streaming?" Idempotent/transactional sinks plus checkpointed offsets; "exactly-once effects," not magical delivery.
- "When is Lambda still justified?" When batch recomputation over massive history is far cheaper/faster in a batch engine than replaying the stream, and you accept the dual code.
- "Micro-batch vs true streaming?" Spark micro-batches trade a little latency for simpler semantics; Flink does true per-event streaming with lower latency and richer state.
Common mistakes.
- Reaching for streaming by default when batch meets the freshness need at far less complexity.
- Not knowing Lambda's core drawback (two synchronized codebases).
- Ignoring late/out-of-order data and exactly-once concerns in the streaming path.
- Treating "real-time" as free, underestimating the operational cost of always-on stateful pipelines.