The response matched the schema exactly. Every required field was present. Every type was correct. The end_date was three weeks before the start_date, and it went straight into the database. On Wednesday morning the data engineer found 847 bookings with impossible date ranges in the warehouse, and someone had to explain in a retro why "we turned JSON mode on" wasn't the same as "we validated the output".
That gap is where this issue lives. Constrained decoding (OpenAI's Structured Outputs, Anthropic's tool-use JSON mode, llama.cpp's grammars) will give you a response that always parses and always matches the shape of your schema. It will not give you a response whose values make sense. The end_date being after the start_date, the account_id existing in your accounts table, the confidence field being between zero and one when the model actually knew what it was looking at: none of that is in the schema. Both can be true, and most teams only ship the first.
Issue 4 covered the observability trace you need to even see bad outputs when they happen. Issue 8 covered the streaming backend that produces them. This issue is the layer between the two: the three-gate output contract that turns "the response parsed" into "the response is safe to persist". Constrained decoding for shape, Pydantic validators for meaning, and one bounded repair retry that feeds the validation errors back to the model. Shown with Pydantic AI's output validators and ModelRetry.
Why valid JSON is not valid data
Constrained decoding works by masking the token distribution at inference time. At each step, the sampler is only allowed to pick tokens that extend a valid prefix of your JSON schema. If the schema says the next field is an integer, non-digit tokens get zero probability. If the schema says a string must match a regex, tokens that would break the regex get zero probability. OpenAI shipped this as Structured Outputs in August 2024 (see Further reading), and the vendor-side guarantee is "100 percent of outputs match the schema". That guarantee is real, and it's narrow.
The guarantee covers syntax and shape. It does not cover values. Three categories of bad data slip through routinely. Numeric ranges: a discount_percent field typed as an integer accepts 500 as happily as 50. Enums you forgot to write: a statusfield typed as a plain string accepts "needs_approval" and "awaiting_approval" from the same model on different days. Cross-field invariants: end_date >= start_date, total == subtotal + tax, shipping_country in available_countries_for(product_id). Every one of these is in the business logic, none is in the JSON schema, and the model's not going to invent the check for you.
There's a second, subtler issue that catches teams moving to structured outputs for the first time. The 2024 paper "Let Me Speak Freely" (see Further reading) measured task performance when models are forced into strict JSON format and found that in some cases, reasoning quality drops versus free-text output on the same question. The intuition is that a strict schema front-loads the format decision over the thinking decision; the model is partly busy producing a legal token at every step and has less distribution left for actual reasoning. The fix isn't to abandon structured outputs; it's to shape the schema so the model has a place to think before it has to commit to an answer. That's one of the schema rules below.
The three-gate output contract
Three gates between the model's response and your database. Each gate catches a different class of error. Skip any one of them and the error class it catches ends up in the warehouse.
Gate 1: Constrained decoding for shape. Use the vendor's structured-output mode. OpenAI's response_format={"type": "json_schema", ...} with strict: true. Anthropic's tool-use with a schema on the tool definition. Pydantic AI wires either of these in behind a result_type on your agent. The gate guarantees the response parses and matches the schema's types. It costs you nothing but a tiny amount of latency on the first request to compile the grammar.
Gate 2: Pydantic validators for meaning. Every field that has a semantic constraint gets a validator. Ranges on numeric fields. Literal types or Enum subclasses on anything that should be a fixed set. model_validator(mode='after') for cross-field invariants. For foreign-key-style fields, a lookup against your actual data store ("does this account_id exist in the accounts table"). The gate catches values that are correctly shaped and semantically wrong.
Gate 3: One bounded repair retry. When gate 2 fails, raise ModelRetry from the validator with the human-readable error. Pydantic AI (and most agent frameworks) will send the error back to the model as a new turn and ask it to fix the specific field. Cap this at exactly one retry per response. The reason for the cap is that unbounded repair loops double or triple your latency and token cost for a problem the model may not actually be able to fix on this turn (bad retrieval, hallucinated ID, prompt ambiguity). Fail closed after one retry and surface the error to the caller.
parses, matches shapeparse failvalues validvalidation errorretry succeedsretry failsModel responseGate 1:constraineddecodingGate 2:PydanticvalidatorsShould not happenwith strict modePersistGate 3:one repair retrySurface structured errorto caller
The three gates in the order they fire. Constrained decoding guarantees shape at the vendor; Pydantic validators enforce meaning in your code; one bounded repair retry gives the model exactly one chance to fix its own output before the caller sees a structured error. Writes to the database only happen on the green path.
The code: Pydantic AI output validators
Pydantic AI is the cleanest way to express the three-gate contract in Python as of mid-2026, though the pattern ports to Instructor, LangChain's output parsers, or hand-rolled code. The snippet below is the complete working shape.
from pydantic import BaseModel, model_validator
from pydantic_ai import Agent, ModelRetry
class Booking(BaseModel):
start_date: date
end_date: date
account_id: str
@model_validator(mode="after")
def _end_after_start(self):
if self.end_date < self.start_date:
raise ModelRetry(
f"end_date {self.end_date} is before start_date "
f"{self.start_date}. Fix the end_date."
)
return self
agent = Agent("openai:gpt-4o-2024-08-06", result_type=Booking, retries=1) Three things to notice. The result_type=Booking plugs the schema into gate 1; the vendor's structured-output mode gets wired up by Pydantic AI automatically. The model_validator is gate 2; it runs after Pydantic has parsed the response into a Booking. Raising ModelRetry with a specific message is what makes gate 3 productive: the model gets the exact error back and often corrects on the first retry. The retries=1 is the cap. Without it, you've built an unbounded loop.
The last piece is the account_id lookup, which isn't shown in the snippet. In production that's a second model_validator that calls accounts.exists(self.account_id) and raises ModelRetry("account_id X does not exist") on miss. This gate catches hallucinated IDs, which still happen in late 2026 at rates that matter.
Schema rules that hold up
The schema itself is a surface the model has to reason over. A badly shaped schema will degrade the reasoning (per the "Let Me Speak Freely" paper) and amplify the gate-2 failure rate. Three rules make the difference in practice.
Evidence fields before answer fields. Put a reasoning field or an evidence field at the top of the schema, and the answer fields after. The model gets to produce the reasoning first, which pays the thinking cost before the format cost, and the answer that comes after is better calibrated. The reasoning field also gives you something to log, which Issue 4 will thank you for when a bad answer lands and you want to know why.
An explicit cannot_determine branch. Every schema that produces a classification, a decision, or an extraction should have an "I don't know" output. A boolean cannot_determine field, or a sum type where one variant is {"status": "cannot_determine", "reason": str}. Without it, the model has no legal token sequence for "not enough information", and it will fabricate rather than refuse. This is the pre-training-origin fabrication from Issue 18, surfaced in the schema.
Shallow nesting. Flat beats nested. Three levels is a yellow flag; five is a red one. Nested objects increase the number of brackets the model has to balance in its token stream, which is a format-tax the "Let Me Speak Freely" dynamic penalises. If the data model wants deep nesting, flatten it to a list of records with a parent key, and reconstruct the tree in your code after validation. Your database is better at hierarchy than the model.
A worked example: the booking schema done right
The opening-scene booking schema was something like {start_date, end_date, account_id} with no reasoning field, no cannot-determine branch, and no cross-field validator. Here's the shape that would have caught the bug on day one.
class Booking(BaseModel):
reasoning: str
status: Literal["book", "cannot_determine"]
start_date: Optional[date] = None
end_date: Optional[date] = None
account_id: Optional[str] = None
cannot_determine_reason: Optional[str] = None
@model_validator(mode="after")
def _check(self):
if self.status == "book":
if not (self.start_date and self.end_date and self.account_id):
raise ModelRetry("book requires start_date, end_date, account_id.")
if self.end_date < self.start_date:
raise ModelRetry("end_date must be >= start_date.")
return self The reasoning field is where the model does the date arithmetic before committing to the fields. The status discriminator forces a legal "I can't tell" output. The model_validator is one place that owns every cross-field invariant. On the Wednesday 847-record bug, the end-before-start rule raises on each bad response, the retry gives the model a specific error to fix, and if the retry still fails the booking lands in a dead-letter queue instead of the warehouse.
Common mistakes
Treating "parses" as "valid". The schema matched. Shipped. The field values were nonsense. This is the entire reason the three-gate contract exists. Add gate 2 before anything persists.
Unbounded repair loops. The validator raises, the agent retries, the retry fails, the agent retries again. Latency goes to seconds, token cost goes to dollars, the model's not actually fixing the problem because the root cause isn't in the format. Cap at one retry. If gate 3 fails, surface the error.
Enum fields typed as plain strings. status: str accepts "needs_approval", "awaiting_approval", "needs-approval", "pending approval", and nineteen other variants over the lifetime of your product. Type it Literal["pending", "approved", "rejected"] or enum.StrEnum. The vendor's constrained decoding will enforce it at gate 1 and the data engineer's dashboard will stop breaking every week.
Deep nesting with no reasoning field. A five-level nested schema and no place for the model to think will produce confidently-shaped, often-wrong answers. Flatten the schema, add a reasoning field at the top, and watch the validator failure rate drop.
The takeaway
JSON mode is on isn't the same as the data is safe. Constrained decoding guarantees shape at the vendor; nothing in the schema guarantees meaning. The three-gate contract (constrained decoding for shape, Pydantic validators for meaning, one bounded repair retry that sends the specific error back to the model) is the small, boring layer that turns a model response into a database write you can defend. Three schema rules make the whole thing cheaper: evidence fields before answer fields, an explicit cannot_determine branch so the model has a legal "I don't know" output, and shallow nesting so the format tax doesn't eat the reasoning. Add them once and the structured-output bugs stop making it to Wednesday morning's warehouse scan.
Production checklist
Turn on the vendor's constrained-output mode on every structured endpoint: OpenAI
response_formatwithstrict: true, Anthropic tool-use with a schema, or the equivalent in your framework.Replace plain string types with
Literal[...]or an enum on every field that should be a fixed set.Add numeric range validators (
conint(ge=0, le=100)style) on every number whose domain is bounded.Add a
model_validator(mode='after')for every cross-field invariant you can name. The booking end-after-start rule is the archetype.For every ID-shaped field, add a validator that looks the ID up against the source of truth and raises
ModelRetrywith the specific missing ID on miss.Set the agent's retry cap to exactly 1. On retry failure, surface a structured error to the caller and record the rejection in the Issue 4 trace.
Add a top-level
reasoningorevidencefield to every schema that drives a decision. Log the reasoning next to the validated output.Add an explicit
cannot_determinebranch to every schema that produces a classification, decision, or extraction. Treatcannot_determineas a first-class downstream signal, not an error.Flatten any schema deeper than three levels. Reconstruct the nested shape in your code after validation.
Add validator-failure counters to the Issue 4 dashboard, broken down by validator name. A spike on a single validator is the earliest signal that the prompt or the retrieval regressed.