Audit Trails

TL;DR: An audit trail logs enough to reconstruct and explain any decision the system made: the input (or a privacy-safe reference), the retrieved context, the model and prompt version, the output, who and when, and any human overrides or guardrail events. It is the backbone of debugging, incident response, compliance (the EU AI Act and regulated domains require traceability), and accountability. The tension is privacy, logs are a sensitive surface, so you redact PII, control access, and set retention. In serious deployments, "why did the model decide that, six months ago?" must be answerable.

What to log

An audit trail goes beyond ordinary observability (which is about operating the system today) to accountability: answering "what happened and why" for one specific decision, possibly long after the model and prompt have both changed. Observability dashboards are append-and-forget aggregates; an audit record is keyed to a single decision and must survive the next deploy.

The non-negotiable fields: input, retrieved sources (for RAG, the actual chunk IDs and scores, not just "we did retrieval"), the exact model and prompt version used (ties to lineage), the output or action, identifiers and timestamps, and human interventions plus guardrail trips. Together these let you replay any decision instead of guessing at it.

A concrete record

One JSON line per decision, written to append-only storage (S3 with object lock, or an immutable table):

{
  "decision_id": "dec_9f2a",
  "ts": "2026-06-19T14:22:31Z",
  "tenant_id": "acme",
  "user_id_hash": "u_7c1e",
  "model": "frontier-llm@2026-04",
  "prompt_version": "support-agent.v17",
  "input_ref": "vault://inputs/dec_9f2a",
  "retrieved": [{"doc": "kb_402#3", "score": 0.81}, {"doc": "kb_119#1", "score": 0.74}],
  "output": "Refund approved per policy 4.2",
  "guardrail_events": ["pii_redactor: 1 email masked"],
  "human_override": null
}

Note input_ref, not the raw input: the prompt may contain a customer's SSN, so the sensitive payload lives in an access-controlled vault and the audit line carries a pointer. The whole record is roughly 400 bytes. At 5M decisions/day that is about 2 GB/day, 730 GB/year before compression, cheap to keep on cold object storage for the retention window.

The privacy tension

Audit logs are a sensitive data surface: they hold PII, prompts, and user content. Balance completeness against exposure with three controls, summarized below.

Set retention deliberately: regulated finance/health often needs years, while a chat product may need 90 days. Log what you need for the purposes above, not indiscriminately.

Why interviewers probe this

In serious deployments, decisions must be explainable and reproducible, and a system with no audit trail cannot meet compliance or debug a harmful output. A strong answer specifies the decision-level fields (input ref, context, model+prompt version, output, who/when, overrides), the uses (debugging, compliance, accountability), and the privacy balance (redact, encrypt, access-control, retain). Recognizing that you log decisions, not just errors, tie them to model version, and store the record append-only, signals governance maturity.

Common misconceptions

• "Logging errors is enough." Audit trails log decisions (including normal ones) so you can explain and reproduce them.
• "Skip the model/prompt version." Without it you cannot reconstruct which model produced a past decision after the next deploy.
• "Log everything in full." Logs hold PII; redact or vault, encrypt, access-control, and set retention.
• "Mutable logs are fine." If logs can be edited, they prove nothing in an incident; use append-only or object-lock storage.

Key takeaways

• An audit trail logs enough to reconstruct and explain any decision: input ref, context, model+prompt version, output, who/when, overrides, guardrail events.
• It underpins debugging, incident response, compliance traceability, and accountability.
• Tie logs to model/prompt version (lineage) and write them append-only so a past decision is reproducible and tamper-evident.
• Balance against privacy: vault/redact PII, encrypt, access-control (and audit access), and set retention.