Jev Capability Atlas

It argues the opposite. The project describes itself as independent, unofficial, and not sponsored by TypeSafe, cites existing benchmark repositories instead of re-running them, and ships real API request logs with the failures left in. One documented case records a 0.90-confidence wrong answer caused by a typo in an answer option.

Calibration is a group statistic, not a per-answer guarantee, and the cited third-party results show it breaking. On an emotion task Jev averaged 0.819 confidence and got 48 percent right, with 16 percent of items giving the correct answer a probability of exactly zero. The failure direction is looking more certain than it is.

Judgments that need knowledge you did not put into the state, and anything irreversible. The axis it keeps returning to is whether the right answer is fully readable from what you supply. Its own context-compaction test kept 0 of 256 results above 0.3 confidence, and the author argues against making that unsupervised or default-on.

No, and it says so, pointing at jev-benchmarks and thaiexam-jev-charts as the places that do rankings. The question here is when the model is strong, when it is weak, and why, so each case suite ships real API logs, methodology, and a report rather than a single score.

Whether the right answer is fully readable from the state you supply, or needs external knowledge that is not in it. Tasks where the signal is self-contained, such as classification, citation support and sarcasm with the cues present, sit on the strong side; comparing against a whole field sits on the weak side.

It separates paraphrase support, where a claim barely overlaps the quote but genuinely supports it, from reversed-meaning overlap, where the text matches almost word for word but one word reverses the meaning. A pure rule or keyword system fails both, so the judgments are not keyword rules.

A history question had one character wrong in the correct option, and Jev chose the wrong answer with 0.90 confidence. Fixing only the typo and re-asking made it stop choosing that option, which shows it will not flag a flawed question and can be confidently wrong when the state is wrong.