Structured outputs, and the bug they cannot fix

A model emits a probability distribution over the next token. Constrained decoding masks out every token the grammar would not allow, renormalises, and samples from what remains.

At each step the model produces a probability for every token.
Constrained decoding zeroes the ones the grammar forbids, then samples.

  schema says: after {"status": we need one of "open" | "closed"
  raw scores:  "open" 0.4   "closed" 0.35   "pending" 0.2   "  " 0.05
  after mask:  "open" 0.53  "closed" 0.47   (everything else -> 0)

The output now parses. Nothing about it is more correct: "pending" was the
truth 20% of the time, and the mask deleted the model's ability to say so.

The mask operates on syntax. It has no access to whether the answer is right, because nothing at that layer knows what right means. What it can do is remove the escape hatch: a model that would have hedged, said "unclear", or produced prose explaining the ambiguity is now forced to pick one of your enum values.

Before structured outputs, a confused model produced malformed JSON, your parser threw, and you found out. Afterwards it produces perfectly valid JSON containing a guess, your parser succeeds, and the guess flows downstream as fact.

This is a strictly worse failure mode, and it is easy to miss because the error rate in your logs went down. It did not: it moved from parse errors, which are loud, to wrong values, which are silent.

// The schema that produces confident nonsense.
{
  "type": "object",
  "properties": {
    "sentiment": { "enum": ["positive", "negative"] },
    "confidence": { "type": "number" }
  },
  "required": ["sentiment", "confidence"]
}

// The same schema, able to represent reality.
{
  "type": "object",
  "properties": {
    "sentiment": { "enum": ["positive", "negative", "neutral", "unclear"] },
    "confidence": { "type": "number" },
    "evidence": { "type": "string" }        // forces it to point at something
  },
  "required": ["sentiment", "confidence", "evidence"]
}

The fix is schema design rather than tooling. If reality includes "neither" and "I cannot tell", the schema has to include them, or the model cannot report them. Adding a required evidence field is the cheap version of the same idea: it is much harder to fabricate a quote than a label.

Reasoning gets squeezed out. A model asked to emit an object starting with "answer" commits before it has worked anything out. Put a reasoning field first in the schema and property order becomes load-bearing, which is a strange thing for a schema to be but is genuinely how it behaves.

Not every schema is supported. Providers restrict what they will enforce: recursion, some composition keywords and open-ended maps are commonly excluded. A schema that validates locally can be rejected at the API, and the workaround usually flattens the model your code wanted.

Confidence numbers are decorative. A confidence float in a schema is a token sequence the model produces because you asked for one. It is not a calibrated probability, and treating it as a threshold gives you arbitrary filtering that feels principled.

Structured outputs are a parsing guarantee. Use them for exactly that, and keep whatever validation you would have written if the response had arrived as free text: range checks, referential checks, and a path for "the model does not know".

The abstraction did not remove the need to verify. It removed the symptom that used to remind you to.