Article
CommunityJev Benchmarks & Evaluations
LLM Judges for Identity Resolution at 4x Lower Cost
Hightouch blog write-up testing whether fast decision models Jev and GPT-6 Luna can grade identity resolution as well as frontier LLMs. Reports out-of-the-box accuracy, calibration differences, and a hybrid Jev plus Opus harness that reaches frontier accuracy at over 4x lower cost.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Both decision models scored 88.8-97.3% accuracy out of the box on the identity resolution grading task.
- A hybrid harness pairing Jev with Claude Opus reaches frontier accuracy at more than 4x lower cost.
- Jev's confidence is described as better calibrated: most of its mistakes come with low confidence, while most of Luna's come with high confidence.
- Decision models are reported as up to 90x cheaper and over 10x faster, at roughly $0.5-$0.10 per 1,000 identities and about 0.15 seconds per answer.
- Frontier models remain more accurate, with decision models trailing Opus by about 6 percentage points on harder benchmarks.
Questions about this entry
Short answers drawn from the linked project or page.
Whether fast, cheap decision models such as Jev and GPT-6 Luna can grade identity resolution results as well as frontier LLMs, and whether a hybrid setup can close the accuracy gap at lower cost.
Both models performed acceptably out of the box, landing between 88.8% and 97.3% accuracy. Luna performed better than Jev out of the box, but the gap disappeared on harder cases.
Jev makes the decision and can ask Claude Opus for help when needed. The write-up reports that this pairing gets close to Opus-level accuracy while costing more than 4x less than using the frontier model alone.
The post reports decision models as up to 90x cheaper and more than 10x faster out of the box, at roughly $0.5-$0.10 per 1,000 identities and about 0.15 seconds per answer.
Most of Jev's mistakes come with low confidence, while most of Luna's mistakes come with high confidence. That difference lets a harness route low-confidence Jev decisions to a stronger model instead of trusting every answer equally.
It was written by Albert Jin, Neil Desai, and Delaney Mackenzie for the Hightouch blog and published on November 4, 2026, under the data and technology teams section.
Frontier models remain more accurate than decision models, which trail Opus by about 6 percentage points on harder benchmarks. The post also examines whether the test set captures enough complex cases and how a confidence threshold is chosen.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Facebook: Intake Router keeps diagnosis out of the answer space at the front desk.
FacebookClinicnotes
X: OpenRouter Decision Model Rankings
X: herdr + jev, an agent router
X: A group chat bot with decision room routing
More Jev resources like this
Controlled benchmark of Jev against Claude Sonnet 5 on one coding-agent decision: given a bug report, which file needs editing? It scores 200 real SWE-bench Verified issues with cost, latency, and confidence-band routing tables.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
CommunityK624/jev-localization-bench
arXiv paper testing whether Jev, a decision model that outputs judgment probabilities, works as a multilingual judge. It compares Jev with generative LLM judges across four annotated benchmarks, three evaluation formats, and up to 47 language varieties.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Hugging Face dataset from TypeSafe AI of customer-service dialogs with per-policy decisions from OpenAI and Anthropic models and a mean-probability consensus field. Parquet, English, 1K–10K rows.
Jev Benchmarks & Evaluations#evaluation#benchmark#routing
Community
Hugging Face dataset of run results for the kieve-jev-fractal-300m decision engine, recording per-benchmark tallies of requests, answered, abstained and errored cases along with scores, latency medians and p95s, and retrieval and cost-aware metrics.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
Community
Benchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
Spring Boot and Spring AI benchmark comparing Java keyword rules, Jev 1.13.0, and Claude Haiku 5.5 on warehouse goods-receipt triage, with a 200-remark synthetic dataset and a Python analysis script.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
Communitymohammedahmadi1990/jev-triage
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.