Design a system to run LLM inference over a billion documents offline, as cheaply as possible.

Batch scoring a billion items is a throughput-and-cost problem, the opposite of low-latency serving. Every choice that hurts latency helps you here. Here is how to design for dollars-per-million-documents.

Updated Aug 2026 · Grounded in real Applied AI Engineer interview loops and written to a senior-engineer editorial bar.