CommunityJev Benchmarks & Evaluations
Open-Jev Judgments on CallScreenBench: Calibrated One-Pass Scam Screening with a Small Language Model (arXiv 2609.23959)
Tests one-pass scam-call screening on 41 held-out scenarios producing 577 per-turn decisions: a three-seed ensemble of LoRA-tuned Qwen3-4B reaches AUROC .974 with calibration error .052 and no false alarms on legitimate calls, at 64.5 ms per decision on one consumer GPU.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Qwen3-4B is LoRA-tuned so the temperature-scaled softmax over two answer-label logits is P(scam).
- 41 held-out CallScreenBench scenarios yield 577 per-turn decisions for evaluation.
- Three-seed ensemble reaches AUROC .974 with calibration error .052 and is non-inferior to an LLM judge at a .02 margin.
- Runs at 64.5 ms per decision on one consumer GPU, 4.9x lower than the same backbone fine-tuned to generate its answer.
- A fine-tuned ModernBERT encoder is not significantly worse, and all callers are synthetic.
Watch out
arXiv preprint submitted September 21, 2026 (cs.CL) by Simiao Ren and eight co-authors. The authors state the recipe was selected with test-set exposure and all callers are synthetic, so the non-inferiority result against MiniMax-M3 should be read with those limits in mind.
Questions about this entry
Short answers drawn from the linked project or page.
Qwen3-4B is LoRA-tuned so that the temperature-scaled softmax over two answer-label logits is P(scam), with no generated text.
41 held-out scenarios producing 577 per-turn decisions, for a screening task that needs a trustworthy probability after every caller turn in milliseconds.
A three-seed ensemble reaches AUROC .974 with calibration error .052 and is non-inferior to an LLM judge (MiniMax-M3) at a pre-registered .02 margin.
64.5 ms per decision on one consumer GPU, which the paper says is 4.9x lower than the same backbone fine-tuned to generate its answer.
The paper reports no false alarms on legitimate calls, and decisions came 1.14 turns earlier under the same hang-up rule.
The paper says the gain is in the readout and calibration, not accuracy: a fine-tuned ModernBERT encoder is not significantly worse.
The recipe was selected with test-set exposure and all callers are synthetic. The authors claim no architectural novelty and present the contribution as the application plus an evaluation reporting calibration, false alarms, and decision timing alongside AUROC.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
YouTube: Kubernetes admission control with a decision model, fail-closed.
Hacker News: SafeDep Gryph plus Jev checks for compromised coding-agent behavior.
2 pointsDetecting Compromised AI Coding Agents with Jev and GryphRead the thread on Hacker NewsHacker News: Show HN: Agent Chaperone screens tool calls and results with Jev before the agent sees them.
sepehrsafari5 pointsShow HN: Agent Chaperone – Screen AI agent tool calls and results with JevRead the thread on Hacker NewsX: Secret detection with Jev, tested
Related terms
Glossary definitions related to this entry.
More Jev resources like this
A benchmark providing exact instance-level class probabilities across three simulated text domains: peer review, covert information transfer, and credit risk. Because the data-generating process is defined, exact probabilities and Bayes error are computable, enabling KL-divergence scoring of reported probabilities.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A pre-registered study of whether jev-1.13.0 satisfies independence of irrelevant alternatives, using 299,463 calls on intent and topic classification. Every added option shifts the log-odds between the gold and strongest wrong label by 0.20–0.52 nats RMS, 1.2 to 3 times run-to-run variation.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A comparison of JEV with LLMs and human coders on seven political science replications. JEV matches or approaches GPT-6 Luna and Qwen3.8-27B on many tasks, but the authors find no cost advantage over GPT-6 Luna at OpenAI's batch prices.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A controlled adversarial comparison of Jev against five GPT and GLM judges and two local classifiers under identical payloads and policies. Jev is the most robust on input moderation and the fastest overall, but detects only 62% of phishing emails and flags the most benign requests.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
An empirical comparison of Jev as a typed judge for retrospective agent-trace security against four generative judges on 5,219 trajectories. Jev averages positive-class F1 77.8 versus 74.1 for GLM-5.2, leading on two of four benchmark collections.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
JevAdvBench is a black-box adversarial benchmark for RLCD models such as Jev, with 812 typed questions over 66 scenarios and 9,744 single-edit attack variants. On jev-1.13.0 an appended unverified opinion flipped 12.1% of decisions and pushed 38% of confident answers below the review threshold.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.