TL;DR: Do not dump the full history into context. Use tiered memory: a short-term buffer of recent turns verbatim, a running summary of older turns (compressed), and a long-term store (vector DB or structured memory) you retrieve from on demand. Keep durable facts (user preferences, decisions) as structured state, not buried in transcript. This bounds tokens, cost, and latency, avoids lost-in-the-middle, and keeps the agent coherent over long sessions.
How to approach it. State the constraint: context windows are finite and not free (cost, latency, and lost-in-the-middle all degrade with length), so for a long session you must decide what to keep in context, not keep everything. Then lay out the tiers and how facts persist as structured state.
A strong answer. The problem: a long conversation or multi-step task accumulates more history than fits (or than is affordable or effective) in the context window. Even with large windows, stuffing everything costs tokens on every turn, raises latency, and suffers lost-in-the-middle (the model attends poorly to buried context). So you manage memory in tiers:
- Short-term (working) memory. Keep the last N turns verbatim, the immediate conversational state the model needs for coherence right now.
- Mid-term: summarization / compression. As turns age out of the buffer, summarize them into a running summary that stays in context (recursive or rolling summarization). You trade fidelity for a bounded token footprint, keeping the gist without the full transcript.
- Long-term memory: retrieval. Store the full history and prior sessions in a vector store and retrieve only the relevant pieces when the current turn needs them (RAG over the conversation). This scales to arbitrarily long histories: context holds recent turns, the summary, and a few retrieved snippets, not everything.
- Structured state for durable facts. Critical, reusable information (user preferences, account details, decisions made, the task goal) should live as explicit structured state, not be left to be re-summarized or re-retrieved from transcript. Update it deliberately so the agent reliably remembers key facts across turns and sessions.
| Tier | What it holds | Mechanism | Failure if misused |
|---|---|---|---|
| Short-term | Last N turns verbatim | In-context buffer | Coherence loss if too small |
| Mid-term | Older turns, compressed | Rolling summary | Lossy: drops details, errors compound |
| Long-term | Full history, prior sessions | Vector retrieval | Stale or irrelevant pulls |
| Structured state | Preferences, decisions, goal | Explicit store, deliberate writes | Drift if not invalidated |
Operational details: decide what is worth remembering (not every turn matters), avoid stale or contradictory memories (update and invalidate), pin the original goal so a long task does not drift, and watch the token budget so memory does not crowd out the actual task. For tool-using agents, persist intermediate results and checkpoints so a long task is resumable.
The defensible framing: bound context with a recent buffer plus summary plus on-demand retrieval, and persist durable facts as structured state, rather than relying on an ever-growing prompt.
Key takeaways.
- Finite context is the constraint; the design question is what to keep, not how to fit everything.
- Four tiers: verbatim recent buffer, rolling summary, retrieved long-term store, structured durable state.
- Precise facts belong in structured state, not in lossy recursive summaries.
- Pin the goal and invalidate stale memories, or the agent drifts and acts on outdated info.
What interviewers probe next.
- "Why not just use a huge context window?" Cost per token every turn, latency, and lost-in-the-middle; long-running agents also exceed even large windows eventually. Retrieval plus summary is cheaper and more reliable.
- "Summarization risks?" Lossy: it can drop details that matter later, and errors compound across recursive summaries; keep critical facts in structured state and retrieve originals when precision is needed.
- "How do you decide what to store long-term?" Salience: durable facts, decisions, user preferences, and task-relevant context; not every chit-chat turn.
- "Avoiding goal drift in a long task?" Pin the original objective in context and periodically re-check actions against it.
Common mistakes.
- Concatenating the entire history every turn, hitting cost, latency, and lost-in-the-middle.
- Relying on summarization for precise facts that should be structured state.
- Never invalidating stale or contradictory memories, so the agent acts on outdated info.
- Letting memory crowd out the task, or losing the original goal over a long session.