Real policies contradict each other. A refund is allowed within 30 days, but not for final-sale items; a support ticket is urgent if the customer is on an enterprise plan, unless the issue is a known outage; a document is approved if it has all required fields, but one of the fields is optional for international orders. The model reads the criteria literally. If the criteria conflict, it will still return a confident answer, and the answer may depend on which sentence it weighted more.
The fix is not to write longer criteria. It is to decide the precedence in code, or to split the judgment until each question is unambiguous.
The model does not resolve policy conflicts for you
The jaggedness notes list contradictory criteria as a known failure mode. The model takes instructions literally, so two rules that point in different directions produce an answer, not an error. The structural-invariants warning is related: complementary probabilities can sum past one, and the same judgment can disagree across primitives. You cannot rely on the output to tell you that the policy was incoherent.
That is why the first step is a policy review, not a prompt rewrite. Write the rules as a table with a precedence order, the way a human reviewer would apply them. If two rules can fire at once and the table does not say which wins, the model is not the problem yet.
Precedence in code, judgment in the model
The composite-scoring pattern shows the general shape: the model returns components, and code applies the weights and the policy. For conflicting rules, the same shape works with gates:
- Ask the model for the facts each rule depends on.
- Apply the precedence table in code.
- Only ask the model for the final judgment when the table leaves a genuine ambiguity.
For example, do not ask “should this be refunded?” over a state with a policy document. Ask is_final_sale as a Noul, days_since_purchase as a Score or a code computation, and is_defective as a Noul. Then apply the rule: final sale wins over the window, defect wins over final sale, and anything else goes to review. The model supplies evidence; the policy stays testable.
The confidence-gated routing pattern is the escalation layer. When the code-applied rules produce “uncertain,” route to a human rather than asking the model to break the tie. The confidence notes are clear that confidence is not accuracy, so a high-confidence answer on a contradictory state is not evidence that the conflict was resolved correctly.
Split the question until it is atomic
The primitives guide recommends atomic questions. Contradictory criteria are often a sign that the question is doing too much. “Is this request eligible for a refund under the policy?” combines several judgments: the purchase window, the product category, the condition, and the customer tier. Ask each one separately, then combine.
The hierarchical classification cookbook is a useful pattern for taxonomies with exceptions. Walk the tree in code, ask the model only about the branch you are in, and keep the exception rules outside the question. The SDE cascade applies the same idea to extraction: narrow the candidate set first, then extract.
When criteria genuinely conflict and you cannot move the precedence into code, make the conflict part of the question. Instead of two rules, present one question with an explicit conflict option and a definition that says what counts. The advanced structure guide allows structured criteria, so you can express the precedence as data rather than prose.
Test the conflicts on purpose
A policy test set that only contains clean cases will not find this failure. Build a small set of deliberate conflicts:
| Case | Expected behavior |
|---|---|
| Final sale, within window, defective | Refund, because defect overrides final sale |
| Final sale, within window, not defective | Reject, because final sale overrides window |
| Enterprise, known outage, low severity | Escalate, because outage policy overrides severity |
| Missing field, international order | Review, because the exception is ambiguous |
Run these cases against the full workflow, not just the model. The workflow evals report four end-to-end workflows rather than isolated prompts, which is the right level for policy conflicts: the failure may be in the branch logic after the model, not in the model's answer.
The TypedBench framing-sensitivity result is a reminder that wording changes the answer even when the policy is stable. If a contradictory policy is phrased two ways, the model may resolve it two ways. Treat the phrasing as part of the policy version.
Keep the policy in one place
The precedence table should live where the code applies it, not in a document that the question text paraphrases. If the policy changes, the table changes, the question version changes, and the test set changes together. That is the only way to keep the model's input and the code's branch logic from drifting apart.
A useful test is to ask whether a new engineer could predict the outcome from the table alone. If the answer depends on a sentence in the prompt that is not in the table, the policy is still split across two systems.
A short checklist
- Write the precedence table before writing the question.
- Ask for facts, not conclusions, when rules conflict.
- Apply precedence and weights in code.
- Use an explicit
conflictorreviewoption when the table cannot decide. - Test deliberate conflicts against the whole workflow.
- Version the policy text and the question together.
- Keep a human path for the cases the table marks uncertain.
Contradictory criteria are not a prompting problem. They are a policy-design problem that the model exposes. Make the precedence explicit and the model becomes a reliable reader of the rules instead of an accidental arbiter of them.