bloom filter

A Bloom filter answers 'have I seen this?' using a few bits per item instead of storing the key, trading a rare false positive for a huge memory win. Here is the build, the sizing math, and where it pays off in dedup and serving.

A hard streaming-systems question: dedup at high throughput with bounded state. The signal is a tiered state design (probabilistic filter in front of durable state) plus watermark-driven eviction. Here is the architecture that does not OOM.

Crawling a few pages is trivial. Crawling the web means a URL frontier with priority, politeness per host, dedup at billions of URLs, and defenses against spider traps and infinite content. Here is the crawler architecture that scales without getting your IPs banned.