watermarks

Pipelines fail and retry; the real question is whether a retry corrupts your data. The signal is idempotent writes (not 'exactly-once delivery') plus watermarks for late data. Here is how to make re-runs safe without double-counting.

A hard streaming-systems question: dedup at high throughput with bounded state. The signal is a tiered state design (probabilistic filter in front of durable state) plus watermark-driven eviction. Here is the architecture that does not OOM.

You cannot wait for all the data, and buffering an unbounded stream will OOM. The signal is windowed joins with watermarks for stream-stream, lookup joins for stream-table, and a clear story for late and out-of-order events.

Events show up minutes or days after they happened, and a window that already closed gives wrong counts. The signal is event-time vs processing-time, watermarks with allowed lateness, and how you correct already-emitted aggregates.