Free lesson · AI Engineering

Structured Output from LLMs

Easy11 min readaillmrag

Structured Output from LLMs

The Regex That Ruined a Quarter

Early in Meta's internal adoption of LLM tooling, a team built a pipeline that extracted action items from meeting transcripts. The model was prompted to "list the action items as JSON." The engineer wrote a regex to pull the JSON out of the response, parsed it, and wrote the results into a task tracker. It worked in every test. It worked in the demo. It shipped.

Over the following weeks it failed roughly 4% of the time - silently. Sometimes the model wrapped its JSON in markdown code fences and the regex grabbed the fence. Sometimes it prefixed the JSON with "Here are the action items you requested:" and the parse started mid-sentence. Sometimes it produced JSON with a trailing comma, which Python's parser rejects. Sometimes it invented a field called "assignee_name" when the schema said "assignee". Each failure dropped a batch of action items on the floor, and because the pipeline swallowed parse errors, nobody noticed until a director asked why tasks from her staff meeting kept vanishing.

The post-mortem conclusion was one sentence long, and it is the thesis of this lesson: free text is not an interface. The moment an LLM's output feeds a program instead of a human, you need a contract - a schema - and you need machinery that enforces it. This lesson builds that machinery, in three levels of increasing rigor, ending with the pattern I consider mandatory for production: schema-constrained generation plus Pydantic validation plus a bounded retry loop.

Why This Problem Exists At All

Remember the generation loop from the last two topics: the model produces tokens by sampling from a probability distribution, one at a time. Nothing in that loop knows what JSON is. When a model produces valid JSON, it is because valid JSON was the most probable continuation - not because anything guaranteed it. Probability is not a contract.

This creates a spectrum of failure modes, and you will see every one of them in production:

Notice the progression: each level of failure is harder to catch than the last. Syntax errors crash loudly. A renamed field fails quietly. A semantically wrong enum value can flow all the way into your database. Your defenses need to match - and they do, in three layers.

Level 1: Prompt-Based Structure (Necessary, Not Sufficient)

The first layer is asking clearly. This sounds trivial; it is not. The difference between a vague and a precise structural instruction changes failure rates by an order of magnitude.

SYSTEM = """You extract invoice data. Respond with ONLY a JSON object,
no markdown fences, no explanation, matching exactly this schema:

{
  "vendor": string,
  "invoice_number": string,
  "total_amount": number,        // numeric, no currency symbol
  "currency": string,            // ISO 4217 code, e.g. "USD"
  "line_items": [
    {"description": string, "amount": number}
  ]
}

If a field is not present in the input, use null. Never invent values."""

Three techniques in that prompt do disproportionate work: showing the exact schema shape (models imitate structure far better than they follow descriptions), the explicit "no markdown fences, no explanation" (kills the two most common wrapper failures), and "never invent values" with a null policy (attacks hallucinated fields at the source).

And connect this back to topic 1.2: extraction runs at temperature 0–0.2. Structured output is the canonical case where variance is pure downside. A creative invoice parser is a broken invoice parser.

But even a perfect prompt at temperature 0 gives you probability, not guarantee. In my experience, well-prompted extraction on a frontier model fails structurally somewhere between 0.5% and 3% of the time depending on input messiness. At 10,000 requests a day, 1% is 100 silent failures a day. Level 1 alone is a demo. Move to Level 2.

Level 2: Provider-Enforced Structure

Both major providers now offer mechanisms that constrain generation itself - the sampling loop is restricted so that only tokens consistent with valid output can be produced.

OpenAI JSON mode (response_format={"type": "json_object"}) guarantees syntactically valid JSON. That is all it guarantees. The JSON can still have the wrong fields, wrong types, wrong everything - it will just be parseable wrongness. JSON mode eliminates row three of the failure table and nothing else.

OpenAI Structured Outputs (response_format with a json_schema) goes much further: you supply a JSON Schema, and the generation is constrained to conform to it. Fields, types, enums - enforced at the token level.

Anthropic's equivalent pattern is tool use: you define a "tool" whose input schema is your desired output schema, and instruct Claude to call it. The tool-call arguments arrive schema-shaped. (This is the same mechanism behind function calling, which gets its own module - here we are borrowing it purely as a structure-enforcement device.)

from openai import OpenAI
from pydantic import BaseModel
from typing import Optional

class LineItem(BaseModel):
    description: str
    amount: float

class Invoice(BaseModel):
    vendor: str
    invoice_number: str
    total_amount: float
    currency: str
    line_items: list[LineItem]
    notes: Optional[str] = None

client = OpenAI()

completion = client.beta.chat.completions.parse(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "Extract invoice data from the user's text."},
        {"role": "user", "content": invoice_text},
    ],
    response_format=Invoice,     # Pydantic model as the schema, enforced
    temperature=0,
)

invoice = completion.choices[0].message.parsed   # already a validated Invoice

The SDK converts the Pydantic model to JSON Schema, the API constrains generation to it, and you get a typed object back. This is the shortest path to reliable structure, and for many applications it is where you stop.

Warning: Schema enforcement guarantees shape, not truth. The model can produce a perfectly schema-valid invoice with a total_amount that appears nowhere in the source text. Constrained generation eliminates structural failures; semantic failures - wrong values in right-shaped fields - remain, and they are the ones that reach your database looking legitimate. Semantic validation is your job, and it is why Level 3 exists even when Level 2 is available.

Level 3: Validate, Retry, Degrade

Level 3 is the application-side machinery that assumes everything upstream can fail - because it can. Structured Outputs has model and feature constraints; you will sometimes use providers or models without it; tool-call arguments occasionally violate their own schema on edge cases; and semantic checks are always yours. The production pattern is a bounded validate-retry loop with error feedback:

import json
from pydantic import BaseModel, ValidationError, field_validator

class Invoice(BaseModel):
    vendor: str
    invoice_number: str
    total_amount: float
    currency: str

    @field_validator("currency")
    @classmethod
    def currency_must_be_iso(cls, v: str) -> str:
        if len(v) != 3 or not v.isupper():
            raise ValueError("currency must be a 3-letter ISO 4217 code")
        return v

    @field_validator("total_amount")
    @classmethod
    def amount_positive(cls, v: float) -> float:
        if v <= 0:
            raise ValueError("total_amount must be positive")
        return v

def extract_invoice(text: str, max_attempts: int = 3) -> Invoice:
    messages = [
        {"role": "system", "content": SYSTEM},   # the Level 1 prompt
        {"role": "user", "content": text},
    ]
    for attempt in range(max_attempts):
        raw = call_llm(messages, temperature=0)   # your topic-1.1 wrapper
        try:
            return Invoice.model_validate_json(_strip_fences(raw))
        except (ValidationError, json.JSONDecodeError) as e:
            # Feed the error back - the model fixes its own output
            messages.append({"role": "assistant", "content": raw})
            messages.append({
                "role": "user",
                "content": f"Your response failed validation:\n{e}\n"
                           "Return ONLY corrected JSON matching the schema."
            })
    raise ExtractionFailed(text_id=hash(text), attempts=max_attempts)

Four design decisions in that loop deserve your attention, because each one is a scar:

The error goes back to the model. This is the single highest-leverage trick in this lesson. Models are remarkably good at fixing their output when shown the specific validation error. In my measurements across several extraction workloads, retry-with-error-feedback recovers 85–95% of first-attempt failures on the very next attempt. A blind retry - same prompt, hope for different sampling - recovers far less, and at temperature 0 may recover nothing, because you are asking the same distribution the same question.

The loop is bounded. Three attempts, then a typed exception. Every retry costs tokens and latency; a model that failed three times with feedback is telling you the input is pathological. Unbounded retry loops on LLM calls are how you convert a bad input into a bad invoice and a bad bill.

Semantic validators ride along. The field_validator checks encode business rules the schema cannot express. This is where "schema-valid but wrong" gets caught - and the validator's error message becomes the corrective feedback. Write validators as if the model will read them, because it will.

Failure is a first-class outcome. ExtractionFailed is not an error to swallow; it is a state to route - to a human review queue, a fallback parser, a dead-letter log. The meeting-transcript team's real sin was not the regex. It was that failure had nowhere to go.

graph TD
    A[Input text] --> B[LLM call, temp 0<br/>schema-guided prompt]
    B --> C{Parse + Pydantic<br/>validation}
    C -->|Valid| D[Typed object → downstream]
    C -->|Invalid| E{Attempts left?}
    E -->|Yes| F[Append validation error<br/>to conversation]
    F --> B
    E -->|No| G[ExtractionFailed →<br/>review queue / fallback]

This flowchart shows the validate-retry loop for structured output. Input text goes to an LLM call at temperature zero with a schema-guided prompt. The output is parsed and validated with Pydantic. If valid, a typed object flows downstream. If invalid and attempts remain, the validation error is appended to the conversation and the model is called again. If attempts are exhausted, a typed ExtractionFailed outcome routes to a review queue or fallback.

The Cost Column

Structure is not free, and you should know where the tokens go. Three line items:

Schemas cost input tokens. A detailed schema in the system prompt, or a JSON Schema attached to Structured Outputs, adds 100–800 input tokens to every call. At small-model prices this is noise; at flagship prices across millions of calls it is a real line item - one more reason extraction belongs on small models.

JSON costs output tokens. Braces, quotes, and repeated key names are all billed tokens. A response as JSON typically runs 20–40% more output tokens than the same information as prose. If you are extracting twenty fields with long names, field naming is literally a cost decision. ("description" is 1 token; verbose keys add up across line items.)

Retries multiply everything. A 2% retry rate is invisible. A 20% retry rate - usually a symptom of a bad prompt or an over-strict validator fighting the model - doubles your effective cost on a fifth of traffic. Log your retry rate as a first-class metric; it is simultaneously a cost signal and a quality signal.

Interview: A common AI Engineering interview scenario: "Design a pipeline that processes 100K customer emails per day into structured support tickets." Strong answers hit, in order: small model at temperature 0, provider-side schema enforcement where available, Pydantic-style semantic validation, bounded retry with error feedback, a dead-letter queue for failures, and retry rate on the monitoring dashboard. That sequence is this lesson - interviewers are testing whether you know free text is not an interface.

Exercise: Harden Your Client

Exercise: Extend your instrumented client from topics 1.1–1.2 into an extraction pipeline:

1. Define a Pydantic model for a domain you know - a job posting (title, company, salary_range, remote: bool, skills: list), a recipe, anything with mixed types.
2. Implement all three levels: schema-in-prompt, provider-enforced structure (Structured Outputs or tool-use pattern), and the validate-retry loop with error feedback.
3. Feed it ten real-world messy inputs (actual job postings from the web). Log attempts-per-success and total token cost per extraction.
4. Challenge: add a semantic validator that the schema cannot express (e.g., salary_range minimum < maximum), force a violation, and watch the error-feedback retry fix it. Capture the conversation - seeing the model repair its own output against your validator is the moment this pattern clicks.
5. Break it on purpose: run the same pipeline at temperature 1.2 and compare your retry rate. You have now measured the temperature-structure relationship from topic 1.2.

Designing Schemas the Model Can Actually Follow

One more skill separates engineers who fight their extraction pipelines from engineers whose pipelines just work: schema design. Your Pydantic model is not only a validator - it is a communication to the model about what you want, and some schemas communicate far better than others. Four rules I apply to every extraction schema, each one bought with debugging time:

Name fields the way the source text talks. If invoices say "Invoice #" and "Amount Due," the fields invoice_number and amount_due will extract more reliably than doc_ref and payable_total. The model is pattern-matching between your schema and the text; do not make it translate vocabulary on top of extracting values. When your internal naming conventions conflict with source vocabulary, extract with source-shaped names and rename in your own code afterward - that translation is deterministic work, and deterministic work should never be delegated to a probabilistic system.

Enums beat free strings everywhere you can afford them. status: Literal["paid", "unpaid", "partial"] closes an entire class of semantic drift that status: str leaves open - "PAID", "settled", "fully paid", "yes." Every field you can constrain to a closed set is a field that cannot surprise you. The corollary: always include an escape value ("other" or "unknown") in the enum, or the model will force-fit genuinely ambiguous inputs into your categories rather than admit uncertainty - a quiet corruption that is far worse than an honest "unknown".

Prefer flat over nested until nesting earns its place. Deeply nested schemas fail more often, and when they fail, the validation errors are harder for the retry loop to act on ("line_items → 3 → adjustments → 0 → amount" is a worse corrective instruction than "amount"). Two levels is comfortable; four is asking for trouble. If your schema is deep because your downstream system is deep, extract flat and reshape in code - same principle as naming.

Use field descriptions as micro-prompts. Pydantic's Field(description=...) flows into the JSON Schema the provider enforces against, which means every description is prompt real estate: Field(description="ISO 4217 code, uppercase, e.g. USD") does prompt-engineering work exactly where the model is deciding that field's value. Dense, example-bearing descriptions on the ambiguous fields are the cheapest accuracy improvement available in this entire lesson.

Schema design is where topic 1.3 quietly becomes prompt engineering - the schema is part of the prompt. Module 4 makes that relationship explicit; for now, treat every schema you write as something the model will read, because it will.

What You Should Take From This Lesson

Free text is not an interface; probability is not a contract. Structure comes in three enforceable layers: precise schema-shaped prompting at temperature 0, provider-side constrained generation where available, and application-side Pydantic validation wrapped in a bounded retry loop that feeds errors back to the model. Schema enforcement buys you shape, never truth - semantic validation stays your job, and failure needs a routed destination, not a swallowed exception. Structure costs tokens; retry rate is a metric.

You can now get reliable, typed data out of the model. The next topic addresses the other half of the user experience equation: those responses take seconds to generate, and users will not wait staring at a spinner. Streaming is how an 8-second response starts feeling instant - and it has its own sharp edges, especially where it collides with everything we just built.

Back to AI Engineering