CommunityJev Benchmarks & Evaluations
Jev for Voice Agent Turn Detection
Veris tests Jev as a semantic turn-detection gate for one Pipecat voice agent from its VAmoS Pro Bench, with labels complete, short, and long. Interruptions fell from 52% to 11% of caller turns across noise conditions, but task completion barely moved and median replies slowed from 2.1 s to 4.6 s.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Turn interruptions fell from 52% to 11% of caller turns overall, significant in every noise condition (all p < 10^-7).
- By condition: clean 31% to 8%, street 28% to 9%, cafe 33% to 8%, TV 64% to 14%.
- 600 simulated calls total: 100 tasks x 3 runs per configuration; verifier passes went from 50/300 to 53/300.
- Median reply time rose from 2.1 s to 4.6 s, mostly from held-open turns; Jev labels return in about 150 ms.
Watch out
Run by Veris AI on one Pipecat agent, comparing Pipecat 1.12 with Jev against 1.8 with a plain 0.8 s silence timer, using hold times they chose. The post states this is not a fair turn-detector comparison.
Questions about this entry
Short answers drawn from the linked project or page.
Yes. Interruptions fell from 52% to 11% of caller turns across all conditions: clean 31% to 8%, street 28% to 9%, cafe 33% to 8%, and TV 64% to 14%, with all drops significant at p < 10^-7.
No. The plain 0.8 s silence timer passed 50 of 300 calls and the Jev-gated agent passed 53 of 300, while median reply time went from 2.1 seconds to 4.6 seconds. TV speech was still transcribed as the caller's.
Complete, short (cut off mid-phrase), and long (paused for something longer, like finding a bill). Jev gets the caller's words so far plus the agent's last line.
In about 150 ms, according to the post, which is why it can sit inside a live voice call as a drop-in turn gate.
600 simulated calls: 100 tasks times three runs in each of the two configurations, with 75 calls per noise condition.
Because the speech recognizer still writes TV speech down as the caller's, and Jev often waits for it to finish. Its TV interruption rate fell from 64% to 14%, still the highest condition.
It tested one Pipecat agent in one configuration with hold times Veris chose, comparing Jev only against a plain silence timer. A fair test would run all turn detectors through the same calls.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Voice-controlled computer use on a Mac
X: A computer assistant that listens
X: An always-on assistant with no wake word
X: Voice control over Fusion 360
More Jev resources like this
3GitHub stars
The first Chinese-language calibration benchmark for Jev-class System One decision models: 378 items and 467 questions across voice routing, e-commerce service and content moderation, with accuracy, ECE, Brier, NLL, AUROC, option-order and zh-CN/zh-TW robustness metrics.
Jev Benchmarks & Evaluations#community#benchmark#calibration
Community
CodyQin/zh-decision-benchPython
CodyQin/zh-decision-benchPython7GitHub stars
Notebook benchmark that compares Jev with open models and specialised tools on five document-pipeline chores: language detection (Choice(12)), page orientation (Choice(4)), RVL-CDIP classification (Choice(16)), bundle splitting and parse-tier triage, with liteparse doing all PDF rendering and OCR work.
Jev Benchmarks & Evaluations#community#evaluation#document-processing
Community
run-llama/jev_vs_ossJupyter Notebook
run-llama/jev_vs_ossJupyter Notebook4GitHub stars
A pre-release Python framework for predictions on mixed text and tabular data: Jev Noul and Score questions turn text columns into numeric features, then a tabular foundation model such as TabPFN learns from labeled rows in context. Offers hosted and fully local backends.
Jev Repos & SDKs#community#tabular#predictions
Community
edamame-labs/tab-jevPython
edamame-labs/tab-jevPython3GitHub stars
A benchmark that steers decision models through 100 key-and-door grid puzzles with one Jev Choice per move, scored against an exact BFS solver; it found Jev picks an optimal move about 90% of the time yet finishes only 57 of 100 games.
Jev Benchmarks & Evaluations#community#benchmark#gridworld
2GitHub stars
Pre-registered benchmark adding Jev 1.13 to Stanford MedHELM's MedHallu results: one Noul asks whether a proposed answer misrepresents its PubMed abstract. Jev scored 92.9% on 1,000 test items at a 204 ms median and USD 0.03 per 1,000 checks through OpenRouter.
Jev Benchmarks & Evaluations#community#medical#hallucination
Community
stperic/jev-medhallu-benchmarkPython
stperic/jev-medhallu-benchmarkPython0GitHub stars
Reproducible benchmark asking which approved agent answers changed meaning after a config edit: one Jev Choice (same, fact_differs, action_differs, specificity_differs) caught all 19 real changes with 13 false alarms across 109 before/after pairs, versus 33 for a three-tier stack and 19 for an LLM judge.
Jev Benchmarks & Evaluations#community#benchmark#regression-testing
From the Jev community
Posts from builders shipping with Jev right now.