User repository on GitHub
Benchmark of TypeSafe's Jev against Sonnet 5, GPT-5 nano and local LLMs on 770 Reddit AITA verdicts: Brier scores, latency and cost
- Forks
- 0
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Benchmarks & Evaluations
Jev on r/AmItheAsshole
A 770-post benchmark that scores Jev, Sonnet 5, GPT-5 nano, and two local models on Reddit verdicts with weighted Brier scores.
- Stars
- 3
- Language
- Python
- Reading time
- 3 min
- Published by
- Community
- Author
- dchristopoulos
- Last verified
- Sep 2026
About this project
Bottom line: In the original seven-setup benchmark, Jev came second, behind Sonnet 5 (Brier score 0.369 vs. 0.344; lower is better). Jev's median call was 6.3× faster than Sonnet's, and 62× cheaper. That's fast, but not the "40x-200x faster" TypeSafe claims. Scored instead against how commenters split, Sonnet's lead is clearer and unadjusted Jev no longer clearly beats guessing from base rates (Part 3).
Quoted from the project's README. Read the full README.
- Languages
- Python
- Capabilities
- Requires API key
Highlights
- The target is the post's verdict flair, not a moral ground truth. 770 posts are from 2025.
- Unadjusted, Jev's weighted Brier was 0.369 against Sonnet 5 at 0.344. Lower is better.
- Jev's median call was 6.3 times faster than Sonnet's and 62 times cheaper. The author paid about $2.75.
- Eight question designs were tried first. None beat asking which verdict the subreddit would reach.
- A logistic adjustment fit on 300 older posts moved Jev to 0.337. Sonnet was not adjusted the same way in that chart.
Quickstart
bash
python -m jevbench.bench --helpWatch out
No license file. Calls go through OpenRouter. The author is unaffiliated with TypeSafe. A later shared adjustment still left Sonnet ahead by 0.007, which the README calls inconclusive.
Questions about this entry
Short answers drawn from the linked project or page.
It loses, narrowly, on the main comparison. Unadjusted, Jev asking directly scored a weighted Brier of 0.369 against Sonnet 5 at 0.344, lower being better. Sonnet's 0.025 lead has a 95% interval of 0.003 to 0.046, outside noise on its own but not after correcting for five comparisons with Jev, so the README calls it borderline.
No, and the author says so. Jev's median call took 0.39 s against Sonnet's 2.46 s, so 6.3x, and it was 62x cheaper. The README attributes the gap to TypeSafe comparing against a model reasoning at its default level, while the chat models in this benchmark did little or no reasoning, which makes them far faster to call.
The target is the post's verdict flair, which the subreddit sets from the top-voted comment, not a moral ground truth, and on 66 of 770 posts the flair disagrees with what most commenters said. The README is blunt that every model missed more than half of the ESH and NAH posts, and that always answering NTA is right 74% of the time.
Not cleanly. Each model ran once, and the README notes that even at temperature 0 a restarted GPT-5 nano run repeated only 64 of 103 answers. The confidence intervals come from resampling posts, so they show how much a score depends on which posts were picked, not run-to-run variance. Local models were 4-bit MLX builds on a single laptop.
It predicts the post's official verdict flair, which r/AmItheAsshole sets from the top-voted comment, not a moral ground truth. On 66 of 770 posts the flair disagrees with what most commenters said, and always answering NTA is right 74% of the time.
The author paid about $2.75 in total and has no affiliation with TypeSafe. Jev asking directly cost $0.037 per 1,000 posts against Sonnet's $2.291, and the calls went through OpenRouter.
Before the benchmark the author tried eight ways of asking, and none was clearly better than asking directly which verdict the subreddit would reach, so the main benchmark uses that direct question. On 200 older posts, ten questions including the direct one got 55.0%, four more than direct, a difference that could be luck at p=0.22.
More Jev resources like this
A fast, typed Python framework from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models: Jev by default, Microsoft Decision-1 on OpenRouter or OpenAI's Decisions API through the same backend. It returns typed scores, thresholds and status, with optional human-label calibration.
Jev Benchmarks & Evaluations#community#github#python
Community
trustifai/typed_evalsPython
trustifai/typed_evalsPythonA sentiment study of whether Jev's stated confidence matches accuracy, comparing raw scores with Platt scaling and isotonic regression.
Jev Benchmarks & Evaluations#community#python#evaluation
Community
AnthusAI/Jev-CalibrationPython
AnthusAI/Jev-CalibrationPythonIndependent pre-registered audit of Jev on Spanish: 19,200 calls over 3,200 paired items on XNLI, PAWS-X, MASSIVE and Belebele. A Spanish state costs 3.0-6.4 accuracy points and roughly doubles ECE on the two hardest tasks, while Spanish instructions change nothing; a CLI reruns the same comparison on your own data.
Jev Benchmarks & Evaluations#community#evaluation#calibration
Community
marcosmartinez/jev-acentoPython
marcosmartinez/jev-acentoPythonA BEIR reranking recipe that asks up to 30 parallel Noul relevance questions in one Jev state, with RESULTS.md against cross-encoders and hosted rerankers.
Jev Benchmarks & Evaluations#community#evaluation#python
Community
hev/rerankerPython
hev/rerankerPythonA benchmark harness and web UI that cross-compares dozens of zero-shot information-extraction and classification systems — extractor encoders, rerankers, and typed-decision engines like Jev — on accuracy, latency, and cost.
Jev Benchmarks & Evaluations#community#python#benchmark
Community
umstek/zero-shot-ie-benchPython
umstek/zero-shot-ie-benchPythonA calibration write-up and scripts for Jev comment and code rules on a Unity project, with ordering.py to compare rule versions on real blocks.
Jev Benchmarks & Evaluations#community#evaluation#code-review
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.