大語言模型回傳有效 JSON，應用程式就可以信任嗎？

Getting an LLM to return valid JSON feels like a milestone. The response parses, the fields exist, and the application can finally do something with the output.

But there’s another question: are those fields actually correct?

Consider a support assistant that turns a customer’s message into a structured request:

{
  "order_id": "ORD-1042",
  "action": "refund",
  "reason": "damaged_item"
}

Everything looks reasonable. Yet the customer might have written: “My item arrived damaged. Can I get a replacement? I don’t want a refund.”

The JSON is valid. The action is wrong.

Three checks worth separating

1. Can the application parse it?

This is the syntax check. Is the response valid JSON, or did it include extra text, broken quotes, or an incomplete object?

2. Does it match the expected structure?

This is the schema check. Are the required fields present? Are values the right types? Is action one of the allowed options?

A schema can reject an unknown action. It cannot, by itself, determine whether an allowed action matches the customer’s intent.

3. Is the result supported and permitted?

This is where application logic matters. Does the order belong to the authenticated customer? Did the customer request this action? Is the order eligible? Does the operation require confirmation?

These checks need information beyond the shape of the model’s response.

Separate interpretation from execution

For a workflow like this, I’d treat the model’s output as a proposed interpretation. The backend would check the order, apply the relevant rules, and ask for clarification when the request is ambiguous.

It also helps to give the model a way to express missing information. A required field with no “unknown” option creates pressure to fill in something, even when the source doesn’t support a confident answer.

For example, an incomplete request could go into a needs_clarification state instead of being forced into refund or replacement.

That state should have a real application path: ask a follow-up question, preserve the existing information, and wait before taking action.

A small test that reveals a lot

Try evaluating the same workflow with:

• A clear request.
• A request that explicitly rejects an action.
• A message containing two different order numbers.
• A request with a missing identifier.
• A customer changing their mind halfway through the conversation.

Check both the extracted fields and what the application does next. A response can pass every formatting check and still lead to the wrong operation.

Structured output makes model responses easier to process. Deciding when to trust those responses remains part of the application’s design.

Have you encountered an LLM response that passed validation but was still wrong in a way that mattered? How did you catch it?