Three Ways to Get JSON Out of a Model

The request side is closed: everything the model receives is contracted, demonstrated, and versioned. Now turn around and face what comes back. Module 2's seam hands you text: string, and for a chat feature that's the product. But most backend LLM work isn't chat. It's extraction, classification, enrichment, routing: calls whose consumer is code, and code needs a shape, not prose.

The obvious move is adding "Respond only with valid JSON matching this format" to the prompt, and it's the move everyone ships first. This lesson is about why that's hope rather than engineering, and about the three real mechanisms that replace it. The database frame carries the whole topic: prompting for JSON is schema-on-read (write whatever, pray at parse time); the mechanisms move you toward schema-on-write, where invalid shapes can't be produced in the first place.

Prompting for JSON is hoping for JSON

An instruction is a clause with imperfect compliance, and "respond only with JSON" fails in ways every team rediscovers: the response wrapped in a markdown code fence; a friendly preamble ("Here's the extracted data:") before the brace; trailing commentary after it; single quotes, trailing commas, JavaScript comments; and the subtler shape failures, a field renamed, an enum value invented, a string where you wanted a number. Compliance rates are high, which is the trap: 98% means a parse failure every fiftieth call, and module 2 taught you that a failure mode without a plan is a 2am improvisation. For a demo, JSON.parse in a try/catch is fine. For a pipeline, "usually parses" is a pager.

The three real mechanisms

JSON mode is the crudest: a request flag (OpenAI's response_format: { type: "json_object" }) that constrains the model to emit syntactically valid JSON. Valid, but of whatever shape the model chooses. No field guarantees, no types, no enums. It eliminates the markdown-fence-and-preamble class of failure and nothing else. Useful when the shape is genuinely free-form; a half-measure when you know what you want.

Structured outputs is the real mechanism: you attach a JSON Schema to the request, and the response conforms to it. Understand why this guarantee is different in kind from a prompt instruction. Module 1's mental model: the model produces a probability distribution over every possible next token. With a schema attached, the provider compiles your schema into a grammar and, at each sampling step, masks every token that would violate it. After {"priority":, the only sampleable tokens are the ones beginning a legal enum value. The model cannot emit an illegal shape for the same reason a typed column with a CHECK constraint cannot hold an illegal row: the write path itself refuses. The guarantee is mechanical, enforced by the sampler, not behavioral, requested from the model. That's the schema-on-write moment.

The forced tool call is the veteran's trick, and it matters for a reason that becomes central next lesson: tool-call arguments are themselves schema-conforming JSON. Define one "tool" whose input schema is the shape you want, force the model to call it, and read the arguments. You never execute anything; the "tool" is a fiction that exists to borrow the tool-calling machinery as a structured-output channel. This predates native structured outputs, and it remains the lowest-common-denominator mechanism when you need one approach that works across providers and older models.

Provider reality check

The two dialects your seam already translates diverge here too, in the same mechanical, enumerable way:

Both dialects accept a subset of JSON Schema, and the subsets differ (recursion depth, string formats, anyOf support). Consult the current pages (OpenAI, Anthropic) before designing anything clever; next lesson argues you shouldn't want clever schemas anyway. When you need one code path across both providers, the forced tool call through your seam is the common denominator, at the cost of the dialect divergence in tool definitions themselves, which is exactly why module 2 told you not to abstract tool calling prematurely.

// Mechanism 2: native structured outputs (OpenAI dialect).
const res = await openai.chat.completions.create({
  model: pins.extraction,
  messages: rendered.messages,
  response_format: {
    type: "json_schema",
    json_schema: { name: "ticket_triage", strict: true, schema: TRIAGE_SCHEMA },
  },
});
 
const msg = res.choices[0].message;
if (msg.refusal) {
  // The schema was not filled; the model declined. A product event,
  // not a parse error: take the refusal branch from module 2.
  return handleRefusal(msg.refusal);
}
const data = JSON.parse(msg.content!); // shape-guaranteed; trust comes later

// Mechanism 3: the forced tool call, the portable trick (Anthropic dialect).
const res = await anthropic.messages.create({
  model: pins.extraction,
  max_tokens: 512,
  tools: [{
    name: "record_triage",
    description: "Record the triage result for a support ticket.",
    input_schema: TRIAGE_SCHEMA,   // same schema, different vehicle
  }],
  tool_choice: { type: "tool", name: "record_triage" }, // forced: no prose allowed
  messages: rendered.messages,
});
 
const call = res.content.find((b) => b.type === "tool_use");
const data = call?.input;  // your JSON, delivered as "arguments"
// Nothing gets executed. The "tool" exists to borrow the machinery.

What "guaranteed" does not cover

The word "guaranteed" in provider docs is doing precise, narrow work: if the model completes a response, the response conforms to the schema. Three doors stay open.

Refusals. A safety-triggered decline can't be expressed inside your triage schema, so it arrives outside the mechanism: OpenAI's refusal field, Anthropic's refusal stop reason. Code that assumes every 200 contains schema-shaped data has reinvented the 200-that-lies. The refusal branch is mandatory.

Truncation. Constrained decoding steers sampling; it doesn't exempt you from max_tokens. Cut the budget short and you get a prefix of valid-so-far JSON with the closing braces never written. The stop-reason discipline from module 2 applies unchanged: check truncated before parsing, size output budgets for the worst-case legal document your schema allows.

Schema-valid garbage. The mechanism guarantees {"customer_id": "CUST-0000"} is shaped right, not that customer exists. Constrained decoding can even mildly encourage confabulation at the margin: when every token must fit the grammar, "some legal value" gets emitted whether or not a good value exists. Schema design that gives the model honest escape hatches (an unknown enum member, nullable fields) is half the fix and next lesson's subject; validating values against reality is the other half and lesson 6's.

Choosing the mechanism

One consequence of the schema riding the request: it's now part of lesson 3's versioned tuple. A schema change is a prompt-version bump and travels through the same review, eval, canary pipeline; the schema also is prompting (the model reads its field names and descriptions), which is precisely where the next lesson picks up: how to design schemas the model fills well, and how the same skill becomes defining your first real tool.

Key takeaways

• Code-consumed output needs a mechanism, not an instruction. "Respond only with JSON" at 98% compliance is a parse failure every fiftieth call, on your hottest path.
• JSON mode guarantees syntax only; structured outputs guarantees your schema via constrained decoding (a grammar mask over sampling: schema-on-write); the forced tool call delivers schema-shaped "arguments" portably, executing nothing.
• The guarantee is mechanical, enforced by the sampler, which is why it's categorically stronger than a prompt clause. But both dialects accept only a subset of JSON Schema, and the subsets differ; check current docs.
• Three doors stay open: refusals arrive outside the schema (dedicated field or stop reason), truncation still applies (check stop reasons before parsing), and schema-valid garbage parses clean. Branch on the first two; lessons 5 and 6 handle the third.
• Default to native structured outputs; use the forced tool call for portability; treat JSON mode as a niche; and version the schema in the prompt tuple, because the schema is also prompting.