The reason to use Jev is not that it classifies text cheaply. Plenty of models do that. The reason is that every answer carries a confidence value, and code can use that value to decide whether to act at all.
That sounds simple and gets subtle fast. This is a practical guide to setting those gates: where thresholds come from, how to keep them honest, and what to log so you can tune them later.
The three-way branch
The confidence pattern uses a voice-banking example that is worth internalizing. One request classifies the intent and reports confidence. Then code branches three ways:
- Below 0.6 confidence on any intent, route to a support agent. The model's uncertainty is the routing signal.
- At 0.6 or above, a low-stakes action like
check_balanceis safe. The worst case is a wrong read-out. - For an irreversible action like
approve_transfer, require more than 0.85 confidence, otherwise ask the user to confirm first.
The point is not those exact numbers. The point is that each action gets its own gate, because each action has its own price for being wrong.
| Action | Cost of a wrong answer | Gate |
|---|---|---|
| Read a balance aloud | Low, user notices immediately | 0.6 |
| Route a support ticket | Medium, human can re-route | 0.6 |
| Approve a transfer | High, funds move | 0.85 plus confirmation |
| Delete a record | Highest, unrecoverable | Human in the loop |
Confidence is not accuracy
Confidence measures the model's certainty in its own answer, not the probability that the answer is correct. The jaggedness list makes this concrete: complementary probabilities can sum past 1 (a refund question once read 0.72 and 0.47 across two formulations), and the same judgment can disagree across primitives. A confidence of 1.0 describes where the probability mass sits, nothing more.
The design consequence: never use confidence as the only protection for an irreversible action. Keep a hard rule, a permission check, or a human next to the model. Confidence decides whether to ask a person; policy decides what a person is allowed to approve.
Pick a gate you can defend
Intent routing adds a second gate worth copying: route below 0.5 intent confidence to a human, and use a separate complexity Score to decide whether a confident request goes to deterministic code, a specialist model, or a person. That gives you two independent reasons to escalate, low confidence and high complexity, instead of one number doing all the work.
When you choose a threshold, write down the two error costs next to it. A false yes and a false no rarely cost the same, and asymmetry is what moves a threshold off 0.5. If a false yes is expensive, raise the gate and accept more human review; if a false no is expensive, lower it and accept more noise.
Measure before you tune
The self-consistency cookbook is the best available template for threshold work. It runs the same 8-question moderation rubric 15 times per condition and reports that Jev's mean probability standard deviation is 0.0098, against 0.0245 to 0.0543 for the LLM distributions it compares. Routing anything below 0.60 probability to an uncertain bucket lifted policy agreement from 90.8% to 99.2%, while still labeling 74.2% of cases automatically.
Two lessons. First, repeatability is not accuracy: run the same case many times and watch whether the confidence moves before you trust a boundary. Second, a gate trades coverage for agreement, and you can measure both. The cookbook even caught Jev flipping its top label across its own repeats, which is exactly the behavior a gate is designed to catch.
Log the inputs to the gate
A threshold is a policy, and policies drift when nobody can replay them. This engineering note on Jev in the agent runtime suggests logging four things together: the state snapshot, the question version, the confidence, and the policy version. With those on hand, an incident review can tell whether the state changed, the question changed, the model changed, or the threshold did.
Pin the model version while you are at it. jev-latest moves; a gate tuned against one release can behave differently against the next. Community walkthroughs like this practical guide pin jev-1.13.0 after tuning for the same reason.
Start here
- Pick one action and one gate.
- Write down the two error costs next to it.
- Log state, question version, confidence, and policy version.
- Run the same labeled cases repeatedly before trusting the boundary.
- Keep a human path for the middle band, and a hard rule for anything irreversible.
