TL;DR: Do not trust a prompt alone to produce valid JSON. Use the strongest constraint available (constrained/grammar-based decoding or the provider's structured-output/function-calling mode that guarantees schema conformance), then still validate the parsed output against your schema and retry or repair on failure. Define a strict schema, handle partial/streaming output, and treat the model as an unreliable producer you must verify.
How to approach it. Frame it as a reliability problem: the model is a stochastic generator, so at scale it will occasionally emit malformed or off-schema output, and a single bad parse can break an agent step or an integration. Lay out the defense from strongest constraint to validation-and-retry, and note where each applies.
A strong answer. The levers, ordered by how strong a guarantee they give:
| Lever | Guarantee | When to use |
|---|---|---|
| Constrained / grammar decoding | Valid by construction at the token level | Self-hosted serving, strict schemas |
| Provider structured-output / function calling | Schema conformance from the API | The production default when available |
| Schema validation + bounded retry | Catches semantic errors and the rare miss | Always, as a backstop |
| Prompt-only | Best-effort, no guarantee | Never alone for a system that breaks on bad input |
Constrained / grammar-based decoding is the most reliable: restrict the decoder so it can only emit tokens consistent with the schema (a JSON grammar or a regex/finite-state constraint), so the output is valid by construction. Provider structured-output mode (pass a JSON Schema and let the API enforce it) and function calling (a tool schema with conforming arguments returned) are the convenient managed form, and the practical default in production because they remove most parse failures.
Then, regardless of the above, parse and validate against your schema in code (types, required fields, enums, ranges). The flow on failure is bounded and explicit:
Operational details that matter: keep the schema strict (required fields, no free-form where an enum will do) so the model has less room to drift; handle streaming carefully (you cannot parse partial JSON, so buffer or use a streaming-tolerant parser); set sensible defaults and a clear error path when retries fail; and log schema-violation rates so you can see degradation. For function-calling agents, validate the tool name and arguments before executing, both for correctness and security (an unvalidated tool call is an injection and error surface).
The defensible framing: make invalid output structurally impossible where you can (constrained decoding / structured mode), and always verify-and-retry in code, because a production integration cannot tolerate "usually valid."
Key takeaways
- Make invalid output impossible by construction with constrained decoding or provider structured-output mode.
- Validate in code anyway: the API guarantees shape, not semantics (an in-range, sane value is your job).
- On failure, retry with the error fed back, bounded, then fall to a safe default and alert; never proceed silently.
- Validate a function call's name, types, and scope before executing it; treat tool args as an injection surface.
What interviewers probe next.
- "Constrained decoding vs JSON mode vs prompting?" Constrained/grammar decoding guarantees validity at the token level; provider JSON/structured mode is the convenient managed form; prompting alone is best-effort and insufficient for production.
- "How do you handle a validation failure?" Bounded retries with the error fed back, then a safe fallback/default and an alert; never silently proceed on malformed output.
- "Streaming structured output?" You cannot parse incomplete JSON; buffer to completion or use incremental/partial-JSON parsing designed for it.
- "Why validate if the API guarantees schema?" Defense in depth and because semantic validity (a field exists vs the value is sane/in-range) is yours to enforce; also tool arguments must be checked before execution.
Common mistakes.
- Relying on a prompt to produce JSON and parsing it with no validation or retry.
- No schema validation in code, so a subtly off-schema response corrupts downstream state.
- Trying to parse partial streamed JSON.
- Executing a function call's arguments without validating name, types, and scope first.