Large Language Models (LLMs) are powerful, but their free-form text responses can be difficult for software to understand. To build reliable applications, you need data in a predictable format. This guide explains how to get reliable LLM structured output in JSON, from the underlying mechanics to production-ready strategies for handling failures.

Key takeaways

  • Structured output forces an LLM to generate data in a machine-readable format like JSON, often making it follow a specific schema.
  • It works through a process called "constrained decoding," where the model is programmatically prevented from generating text that would break the desired structure.
  • Methods for getting structured data form a "reliability ladder": simple prompting is brittle, JSON mode ensures valid syntax, and schema-constrained output enforces specific fields and types.
  • A valid JSON schema does not guarantee correct content. Always perform semantic validation, like checking business logic, on the data the model returns.
  • Production-ready code must handle failures, such as the model refusing to output the object, the response being cut off, or API errors.

What Is Structured Output in an LLM?

Structured output is a method for forcing an LLM to return data in a machine-readable format that software can directly use, with JSON being the most common format (Structured model outputs). While JSON is most common, the underlying technique of constrained decoding can be applied to other formats, provided they can be defined by a formal grammar. It makes model responses predictable and reliable, which is essential for building applications. Instead of trying to parse inconsistent natural language, your application receives a clean, structured object.

Imagine you want to extract a user's name and email from a support ticket. An unstructured response might be: "The user, John Doe, said his email is john.doe@example.com."

Parsing this sentence is fragile. If the model says "John Doe's email is john.doe@example.com", your code might break.

With structured output, you can ask for this instead:

JSON
{
  "name": "John Doe",
  "email": "john.doe@example.com"
}

This JSON object is easy for any programming language to parse.

It's critical to distinguish between two levels of correctness:

  1. Syntactic Validity: The output is perfectly formed JSON. It has correctly matched braces, quoted keys, and proper commas.
  2. Semantic Correctness: The values inside the JSON are accurate and make sense for your application.

Modern structured output techniques guarantee syntactic validity, but they do not guarantee semantic correctness (Constrained Decoding Eliminates Structural Failures in Small LLMs...). The model can still hallucinate a fake email address that fits perfectly into a valid JSON structure.

How Does Constrained Decoding Guarantee Valid JSON?

Constrained decoding ensures an LLM produces syntactically valid output by restricting which tokens it can generate at each step. The inference engine uses a schema you provide to create a grammar, then filters the model's choices to only allow tokens that follow that grammar, guaranteeing the final text is well-formed (The Hidden Cost of Structured Generation in LLMs). This method is highly effective; one evaluation across several small models found that constrained decoding increased schema validity from a range of 78.6–92.9% to a perfect 100% (Constrained Decoding Eliminates Structural Failures in Small LLMs...). Other research reported improvements of up to 24 percentage points in strict structured accuracy over standard methods (The Hidden Cost of Structured Generation in LLMs).

Here is a step-by-step breakdown of how it works:

  1. Provide a Schema: You send your prompt along with a schema that defines the desired JSON structure. This is often a JSON Schema, a standard for describing the structure of JSON data.

  2. Create a Grammar: The LLM's inference engine converts your schema into a formal grammar or a state machine. This grammar represents all possible sequences of tokens that would result in valid JSON conforming to your schema.

  3. Calculate Next-Token Probabilities: At each step of generation, the LLM does what it normally does: it predicts a probability distribution over its entire vocabulary for the very next token.

  4. Apply a Mask: Before choosing a token, the inference engine applies a "mask." It looks at the text generated so far and the grammar rules. It then sets the probability of all tokens that would violate the grammar to zero.

  5. Sample from Allowed Tokens: The model then samples a token only from the remaining, unmasked options. This choice is guaranteed to keep the output on a path toward a valid, complete JSON object.

This process repeats until the model generates a complete object that satisfies the grammar.

An Illustrative Example

Imagine the model needs to generate an object like {"name": "Alice"}.

  • Start: The model has generated an opening brace {.
  • Grammar Rule: According to JSON syntax, the only valid tokens that can follow a { are a double quote " (to start a key) or a closing brace } (for an empty object).
  • Masking: The engine masks all tokens except " and }. It forbids the model from generating an A, a comma, or any other illegal character.
  • Sampling: The model chooses " from the allowed tokens.
  • Next Step: The generated text is now {". The grammar now requires characters for the key. After the key name and the colon :, the grammar will again require a " to start the string value.

This step-by-step enforcement makes it impossible for the model to produce a syntactically invalid result like {'name': Alice}. Unlike many programming language objects, JSON syntax strictly requires double quotes for both property names and string values, a rule that constrained decoding enforces automatically.

Choosing Your Method: The Reliability Ladder

Not all structured output methods are equal. They exist on a ladder of increasing reliability. As you move up the ladder, you get stronger guarantees, but the setup may become more specific to a particular model or API.

Method Guarantee Level Best For... Common Pitfalls
Simple Prompting None Quick, informal experiments and prototyping. Brittle; often produces malformed JSON, extra text, or omits required fields.
JSON Mode Syntactic Getting parseable JSON when the exact structure is simple or flexible. Does not enforce specific fields, types, or enums. The structure can vary between calls (Structured model outputs).
Schema-Constrained Structural Reliable data extraction, classification, and routing where specific fields are required. Can be more complex to set up; provider support for JSON Schema features varies (Function Calling in the OpenAI API).
Strict Tool/Function Calling Structural Building agents and workflows that execute code based on model output. Tightly coupled to an API's tool-use framework. See our guide on LLM Tool Calling: How It Works and How to Use It.

For most production applications that need reliable data, Schema-Constrained Output is the recommended choice.

A Practical Example: From Simple Schema to Production-Ready

Let's walk through designing a JSON schema for a real-world task: extracting user information from a support ticket. We'll start with a loose schema and progressively make it more robust.

The task is to extract a user's name, preferred contact method, and user ID from the text: "Please contact me, Jane Roe, about ticket #12345. My user ID is U-98765 and email is best."

Version 1: A Permissive Schema

A beginner might start with a simple schema that lists the desired fields as optional strings.

JSON
{
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "contact_method": { "type": "string" },
    "user_id": { "type": "string" }
  }
}
  • Good: It's simple and communicates the goal.
  • Bad: It doesn't require any fields, so the model could return an empty object {}. It also allows any string for contact_method, like "email is best", which isn't a clean value.

Version 2: A More Robust Schema

Now, let's add constraints to improve reliability. We'll make fields required and limit the choices for contact_method.

JSON
{
  "type": "object",
  "properties": {
    "name": {
      "type": "string",
      "description": "The full name of the user."
    },
    "contact_method": {
      "type": "string",
      "enum": ["email", "phone", "sms"],
      "description": "The user's preferred contact method."
    },
    "user_id": {
      "type": "string",
      "description": "The user's unique identifier."
    }
  },
  "required": ["name", "user_id"]
}
  • Better: name and user_id are now mandatory. contact_method is restricted to one of three clean values. The description fields help the model understand the purpose of each field.

Version 3: A Production-Ready Schema

For production, we want to be even stricter to prevent unexpected data and validate formats.

JSON
{
  "type": "object",
  "properties": {
    "name": {
      "type": "string",
      "description": "The full name of the user."
    },
    "contact_method": {
      "type": "string",
      "enum": ["email", "phone", "sms"],
      "description": "The user's preferred contact method."
    },
    "user_id": {
      "type": "string",
      "description": "The user's unique identifier, starting with 'U-'.",
      "pattern": "^U-[0-9]{5,}$"
    }
  },
  "required": ["name", "user_id"],
  "additionalProperties": false
}
  • Best: additionalProperties: false prevents the model from adding extra, unexpected fields. The pattern for user_id enforces a specific format (e.g., "U-98765"), providing another layer of validation at the schema level.

By making your schema more specific, you reduce the model's ambiguity and increase the reliability of its output.

What Are the Common Mistakes and Limits?

While powerful, structured output is not a silver bullet. Beginners often make incorrect assumptions about its guarantees. Here are common mistakes and fundamental limitations to keep in mind.

  • Mistake: Assuming valid JSON means correct content. The single most important limitation is that a schema only constrains the shape of the data, not its factual accuracy (Constrained Decoding Eliminates Structural Failures in Small LLMs...). The model can still hallucinate values that fit the schema. Always validate the content.
  • Mistake: Confusing JSON mode with schema enforcement. JSON mode guarantees the output is syntactically valid JSON, but it does not enforce a specific structure, like required fields or data types (Structured model outputs).
  • Mistake: Assuming consistent JSON Schema support. API providers may only support a subset of the JSON Schema specification. Complex features like conditional subschemas, complex regular expressions, or default values may not work as expected across different models and APIs.
  • Mistake: Forgetting application-level validation. Your code should always be the final arbiter. After receiving a schema-compliant response, check business rules. For example, if you extract start_date and end_date, your code must verify that start_date comes before end_date.
  • Mistake: Building overly complex schemas. Schemas that are extremely large or deeply recursive can hit provider limits, increase latency, or use an excessive number of tokens. Keep schemas as simple as possible to achieve the task.
  • Mistake: Using bounded retries incorrectly. Simply retrying a failed request with the same prompt and schema is often ineffective. A better retry strategy involves changing the prompt, simplifying the schema, or switching to a different model.
  • Limit: The model cannot invent missing information. If you ask for a user's phone number but the input text only contains an email, a schema cannot force the model to find a phone number. It will likely omit the field or (if allowed) use a null value.
  • Limit: Constrained decoding can produce awkward results. Sometimes, the most natural or probable sequence of words for a model might violate the schema. Constrained decoding can force it down a less probable path, resulting in output that is structurally valid but semantically awkward (The Hidden Cost of Structured Generation in LLMs).

How to Handle Failures and Refusals

Even with schema constraints, API calls can fail. A production system must anticipate and handle errors gracefully. This includes network issues, API errors, validation failures, and cases where the model refuses to comply.

The following Python example uses placeholder names for an API library (your_api_library) and its exceptions to illustrate a robust error-handling pattern.

Python
import json
import jsonschema
# Fictional library; replace with your actual API client (e.g., openai, anthropic)
from your_api_library import call_llm_with_structured_output, YourApiException

# Our expected schema (from the "Production-Ready" example)
USER_SCHEMA = {
    "type": "object",
    "properties": {
        "name": {"type": "string", "description": "The full name of the user."},
        "contact_method": {"type": "string", "enum": ["email", "phone", "sms"]},
        "user_id": {"type": "string", "pattern": "^U-[0-9]{5,}$"}
    },
    "required": ["name", "user_id"],
    "additionalProperties": False
}

def extract_user_data(text: str, retries: int = 2) -> dict | None:
    for attempt in range(retries):
        try:
            # 1. Handle API and Network Errors
            response_text = call_llm_with_structured_output(
                prompt=f"Extract user info from: {text}",
                json_schema=USER_SCHEMA
            )

            # 2. Handle Refusals or Empty Responses
            if not response_text or response_text.strip() == "":
                print("Warning: Model returned an empty response.")
                continue # Retry

            # 3. Handle JSON Parsing Errors (e.g., from a truncated response)
            data = json.loads(response_text)

            # 4. Handle Schema Validation Errors
            jsonschema.validate(instance=data, schema=USER_SCHEMA)
            
            # 5. Handle Semantic Validation Errors (business logic)
            if "name" in data and len(data["name"].split()) < 2:
                print("Semantic Error: Name appears to be incomplete.")
                return None # Or retry with a modified prompt

            return data

        except YourApiException as e:
            print(f"API Error on attempt {attempt+1}: {e}")
            # Could add exponential backoff here
        except json.JSONDecodeError as e:
            print(f"JSON Parsing Error on attempt {attempt+1}: {e}")
        except jsonschema.ValidationError as e:
            print(f"Schema Validation Error: {e.message}")
            return None # Schema errors are unlikely to be fixed by a simple retry
        except Exception as e:
            print(f"An unexpected error occurred: {e}")

    print("Failed to get valid data after all retries.")
    return None

This code explicitly handles:

  1. API/Network errors with a retry loop.
  2. Model refusals by checking for an empty response.
  3. Parsing errors in case the response is cut off or malformed despite guarantees.
  4. Schema validation as a final check.
  5. Semantic checks specific to the application's logic.

When Should You Not Rely on Structured Output?

Structured output is a powerful tool, but it's not always the right one. Forcing a rigid structure can sometimes be counterproductive.

Avoid using it in these scenarios:

  • For highly creative or stylistic tasks. When writing poetry, brainstorming ideas, or drafting marketing copy, the constraints of a schema can stifle creativity and produce stilted prose.
  • When the output structure is unknown. During exploratory data analysis or when summarizing a document with an unpredictable structure, forcing data into predefined fields can lose critical nuance. Free-form text is often better.
  • When nuance is the primary goal. Summarizing complex legal or medical documents into a few rigid fields is risky. You may lose important context, exceptions, and qualifying statements that are better captured in a natural language summary.

What to learn next

  • Dive deeper into Tool and Function Calling. Many advanced AI applications use structured output as the foundation for giving models tools they can use. Learn more in our complete guide to LLM Tool Calling.
  • Understand Prompt Injection. When you extract data from user input and place it into a structured format, you must be aware of security risks like prompt injection attacks.
  • Explore Advanced Schema Design. Learn about more advanced JSON Schema features like conditional subschemas (if/then/else), complex array validation, and combining schemas with allOf or oneOf to handle more complex data structures.

Sources and further reading