Structured Outputs

What Is Structured Outputs?

Structured outputs is a technique that forces a language model's response to conform to a developer-supplied schema, typically JSON Schema, so downstream code can parse it without defensive guessing. Instead of hoping the model formats its answer correctly, the API constrains generation so only schema-valid output is possible.

Key Takeaways

• The guarantee is syntactic, enforced at generation time. Constrained decoding masks any token that would violate the schema, so the output always parses and always matches the declared shape.
• Valid structure does not mean correct content. The model can still put a wrong value in a perfectly typed field, so evals and validation of meaning remain your job.
• Structured outputs is what makes LLMs composable with ordinary software. Pipelines, databases, and UIs consume model output as data instead of parsing prose.
• The same machinery powers reliable tool calling, since a tool call is itself a schema-constrained object.
• All major providers ship it in 2026, generally accepting JSON Schema directly or via typed bindings like Pydantic and Zod in the SDKs.

How It Works

A language model generates one token at a time by sampling from a probability distribution over its vocabulary. Constrained decoding intervenes in that step. The provider compiles your schema into a grammar, and at each position the engine masks every token that could not begin a valid continuation. If the schema says the next thing must be a key called "severity" whose value is one of three enum strings, tokens that would produce anything else are simply unavailable to sample. The model keeps choosing among what remains, so the result is fluent and schema-valid at the same time.

Developers usually work a level above this. You define a type in your language, a Pydantic model in Python or a Zod schema in TypeScript, and the SDK converts it to JSON Schema, sends it with the request, and deserializes the guaranteed-valid response back into a typed object. That closes the loop that used to be the fragile part of LLM engineering: no regex extraction from markdown fences, no retry loops for truncated JSON, no try/except around every parse.

The practical caveats are about semantics and expressiveness. Schema conformance says nothing about truth, so a model can return a confidently wrong value in the right field, and an over-constrained schema can even encourage that by forcing an answer where "unknown" was honest. Good schemas leave room for uncertainty with optional or nullable fields. Provider support for JSON Schema is also partial: certain keywords and deeply recursive shapes may be rejected or ignored, so schemas are best kept flat and explicit.

Example

A team builds a triage pipeline that reads inbound bug reports and files tickets. They define a schema: title (string, max 80 chars), severity (enum: critical, major, minor), component (enum of twelve services), reproduction_steps (array of strings), duplicate_of (nullable ticket ID). Every report goes through the model with that schema attached, and the output drops straight into the ticketing API with no parsing layer. Before structured outputs, the same pipeline broke weekly on markdown-wrapped JSON and invented severity labels like "high-ish". After, malformed output disappeared as a failure class, and the remaining errors were semantic ones, like misjudged severity, which they now catch with a small eval suite.

What People Get Wrong

The recurring mistake is reading "guaranteed valid" as "guaranteed right". Structured outputs eliminates format failures, and teams then quietly stop checking content because the pipeline never crashes anymore. That trades loud failures for silent ones. A hallucinated component name is impossible when the field is an enum, but a wrong choice among valid enum values is not, and it flows downstream looking authoritative. Keep semantic validation and evals in place; the schema handles shape, nothing else.

JSON mode

JSON mode is the predecessor and weaker sibling of structured outputs, and many APIs still offer both. In JSON mode the model is constrained to emit syntactically valid JSON, but no schema is enforced: keys can be missing, renamed, or extra, and types can drift between calls. It solved the parse-error problem while leaving the shape problem open, which meant application code still validated everything after the fact. The gap it papers over is real: a 2024 StructuredRAG study measured an average success rate of 82.55% across 24 experiments testing JSON response formatting, with per-task performance ranging from 0% to 100% [1]. Structured outputs subsumes it by enforcing your exact schema during decoding. In 2026, JSON mode mainly survives for quick exploratory work where defining a schema is not worth the effort, and provider documentation steers production use toward schema-enforced outputs. If you are validating a JSON-mode response against a schema and retrying on failure, you are hand-rolling what structured outputs already does.

FAQ

Does using structured outputs hurt response quality? Heavy constraints can interact with reasoning, since the model must commit to a rigid shape while thinking. A 2024 study of format restrictions found GPT-3.5-Turbo's GSM8K accuracy fell from 76.60% in natural language to 49.25% when forced into JSON, a 27-point drop [2]. A common mitigation is including a free-text field for reasoning ahead of the answer fields, or letting the model reason in a first pass and extracting structure in a second.

Can I use structured outputs for streaming responses? Yes. Providers stream schema-constrained tokens like any others, and SDKs can surface partially populated objects as fields complete. The consuming code just has to tolerate incomplete objects until the stream ends.

When should I use tool calling instead of structured outputs? Use tool calling when the model should decide whether and which function to invoke as part of an agent loop. Use structured outputs when you always want exactly one response in a known shape, such as extraction, classification, or report generation.

Sources

1. arXiv. "StructuredRAG: JSON Response Formatting with Large Language Models, success rates across 24 experiments." https://arxiv.org/abs/2408.11061. Accessed August 2026.
2. arXiv. "Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models, GSM8K accuracy under JSON formatting." https://arxiv.org/html/2408.02442v1. Accessed August 2026.