deduplication

Fine-tuning is mostly a data problem, not a hyperparameter one. The signal is naming what makes a good set (quality, diversity, format, dedup) and arguing why a few thousand clean examples beat a million noisy ones.

Top-k by pure similarity often hands the model five paraphrases of the same paragraph, starving it of the other facts the question needs. The fix is ranking for relevance and novelty at once. Here is how.

Comparing every pair of a million documents is a trillion comparisons. MinHash estimates Jaccard similarity from a tiny signature, and LSH turns dedup into a near-linear scan. Here is the implementation.

A Bloom filter answers 'have I seen this?' using a few bits per item instead of storing the key, trading a rare false positive for a huge memory win. Here is the build, the sizing math, and where it pays off in dedup and serving.

A grounded systems-coding screen: find files with identical content across a tree. The naive all-pairs hash is wasteful. The signal is the size prefilter and a cheap-hash gate before the full hash. Here is the layered approach.

A hard streaming-systems question: dedup at high throughput with bounded state. The signal is a tiered state design (probabilistic filter in front of durable state) plus watermark-driven eviction. Here is the architecture that does not OOM.

Dedup is a daily data-engineering task and a common screen. The signal is ROW_NUMBER to keep the right record (not just DISTINCT), defining what 'duplicate' actually means, and handling near-duplicates at scale.

Anyone can say 'filter and dedup.' The signal is the funnel in cost order, the MinHash/LSH banding math, and knowing the shuffle at billions of documents is what actually costs you, plus benchmark decontamination people forget until their eval numbers get challenged.

Crawling a few pages is trivial. Crawling the web means a URL frontier with priority, politeness per host, dedup at billions of URLs, and defenses against spider traps and infinite content. Here is the crawler architecture that scales without getting your IPs banned.

Large models memorize and can be prompted to emit training examples verbatim, a real privacy and copyright liability. The defenses span data, training, and output. Here is the layered answer.