Jevcal

No, and it cannot. The README contains no Jev performance numbers on purpose because TypeSafe's customer agreement restricts publishing them, and jevcal ships no leaderboard and publishes nothing. The demo output you see first is from a built-in simulator that is deliberately overconfident, not from Jev. The argument is that nobody can hand you trustworthy numbers for your use case, so you measure it privately yourself.

Expect it not to hold, and the tool says so rather than hiding it. A threshold is only as good as its sample, and the README puts the floor at roughly a hundred labeled rows per question. Thresholds are picked on one half of your data and verified on the other half; if one fails to hold, jevcal reports that instead of the flattering number. --conservative tightens this further to the 95% lower confidence bound.

You need real examples either way, and labels can be bootstrapped. The dataset is JSONL with a few hundred rows of state and labels. If you have none, jevcal label fills the gaps with an LLM teacher (anthropic, openai, openrouter, or your local claude-cli login) and never overwrites labels you already have, but those labels are a teacher signal rather than ground truth, so spot-check a sample.

That is exactly what check exists to catch, since jev-latest is an alias that moves on every release. It exits 1 when accepted accuracy drops below target, when coverage falls and more traffic escalates, and, with --strict, when the model version that answered is not the one the thresholds were tuned on. The API rounds probabilities to two decimals and identical requests occasionally differ, so check carries tolerances you should not zero out.

It flags question wording the model is known to handle badly: negations, counting, dates, compound questions and overlapping options, so you can fix wording before spending a request.

Run the same questions through a text LLM with --provider llm --llm openai:gpt-...; accepted specs include anthropic, openai (honouring OPENAI_BASE_URL), openrouter and claude-cli, which uses your local Claude Code login.

The shipped Cascade reads decisions.lock.json and, with no fallback configured, fails closed: low-confidence answers come back as None rather than a guess, and each decision reports its source as jev, fallback or unresolved.