Did the Halv workflow solve more tasks than vanilla Codex?

No. Both arms passed 25 of 42 selected runs. Halv gained passes on ArcadeDB-4281 and HugeGraph-3037 but passed only 1 of 3 Perry-3982 runs against vanilla's 3 of 3, so aggregate equality hides per-task differences.

Does the 57.1% figure include every benchmark attempt?

No. It compares selected valid-run costs of $144.67 versus $337.50. Excluded are 13 infrastructure-invalid attempts with $25.53 in known cost plus nine unpriced attempts, and 10 runs from the rolled-back policy experiment.

How is that cost saving possible with more tokens?

Halv used 3.49x more tokens, 170.1M against 48.8M, because it spent them on cheaper models. Cost per verifier pass fell from $13.50 to $5.79, and the saved money sits mostly in routing decisions rather than in using less of anything.

Where did Halv's recorded cost go?

L1 workers took $76.03 or 52.6%, the Astra coordinator $64.50 or 44.6%, L2 workers $3.07 or 2.1%, and the investigation stage $1.06 or 0.7%. The L2 layer appeared in only 4 of the 42 Halv runs, since most used the investigation and L1 stages without adding another delegation level.

What role does JEV actually play in the workflow?

It selects the L1 worker model and reasoning effort after Astra assigns an investigation to Luna 6 with medium reasoning and the investigation informs the task brief. L1 can then hand bounded work to a cheaper L2 worker, and Astra reviews and integrates. Every selected worker used the Codex lane.

How strong is the claim here?

It is Halv AI's own benchmark campaign, an interim checkpoint covering 14 of 111 planned tasks, and a workflow comparison that changed routing, context tools and coordination policy together. It changed worker models rather than isolating one component, the recorded costs are token-based estimates rather than invoices, and the post says so itself. JEV agent selection is included in the Halv subscription, so no separate fee is allocated across runs.

What else did the Halv workflow enable besides JEV routing?

Crux 0.10.3 for code navigation, RTK to filter command output, and Headroom to process model context, plus an external benchmark monitor that supplied status without routine progress updates. Because all of these changed together with model routing and coordination policy, the experiment cannot isolate how much of the saving came from JEV.

CommunityJev Benchmarks & Evaluations

Inside Halv's 57.1% Cost Reduction: SWE-rebench Data

Halv reports a paired SWE-rebench comparison where its workflow, an Astra coordinator, a Luna investigation stage, and JEV selecting worker models and reasoning effort, recorded $144.67 in model cost versus $337.50 for vanilla Codex across 42 runs, with both arms passing 25 verifier runs.

See highlights

Open halv.ai

Format: Article
Published by: Community
Last verified: Oct 2026

Tagscommunity benchmark routing cost

Works forJev for Routing Jev for Code

Used inJev for Software & DevTools

Highlights

42 selected runs (14 tasks x 3 repetitions): both arms passed 25 runs, Halv cost $144.67 versus $337.50 for vanilla Codex.
Halv used 3.49x more tokens (170.1M versus 48.8M); cost split: coordinator $64.50, L1 workers $76.03, L2 workers $3.07, investigation $1.06.
ArcadeDB-4455: 81.7% lower cost ($14.97 versus $81.85) with 3/3 passes in both arms; Perry-3982 regressed to 1/3 passes from 3/3.
The interim checkpoint covers 14 of 111 planned tasks; 13 infrastructure-invalid attempts and a 10-run rolled-back policy experiment are excluded.

Watch out

Published by Halv AI on its own benchmark campaign, a workflow comparison that changed routing, context tools, and coordination policy together, so it cannot isolate JEV's contribution. Recorded costs are token-based estimates, not invoices.

Questions about this entry

Short answers drawn from the linked project or page.

Reactions & coverage

Posts, threads, and videos about this entry from around the web.

X: Hard-coded rules moved to Jev

YouTube: Joe Maddalone tries Jev as a routing and filtering step, with a transcript-analyzer demo.

Facebook: Intake Router keeps diagnosis out of the answer space at the front desk.

FacebookClinicnotes

YouTube: Jev explained for routing and judging, with a comparison against LLMs.

Related terms

Glossary definitions related to this entry.

More Jev resources like this

jev-eval

Third-party TypeScript check that compares Jev (typesafe-ai/jev) against openai/gpt-4o-mini and anthropic/claude-sonnet-4.5 under identical conditions, routing 60 synthetic booking inquiries in four languages for a photo-shoot service, all through one Vercel AI Gateway key.

Jev Benchmarks & Evaluations#community#benchmark#comparison

Community

Shogo-nfrealmusic/jev-evalTypeScript

SREGym

338

SREGym stands up live Kubernetes clusters so SRE agents can be evaluated against 90 fault scenarios, including OS-level faults and metastable and concurrent failures, in containers that cannot see the grading logic.

Jev Benchmarks & Evaluations#community#jevbest

Community

SREGym/SREGymPython

typesafe-jev-calibrate-for-code-review

A calibration write-up and scripts for Jev comment and code rules on a Unity project, with ordering.py to compare rule versions on real blocks.

Jev Benchmarks & Evaluations#community#evaluation#code-review

Community

Selmar/typesafe-jev-calibrate-for-code-reviewPython

Jev Synthetic Survey

Independent evaluation running Jev jev-1.13.0 and GPT-4.1 as the same 300 Twin-2K-500 synthetic survey respondents across 24,596 paired cells, finding that asking a yes/no item as a Noul rather than a Choice moved results more than the model gap, at a thirty-fourth of the cost.

Jev Benchmarks & Evaluations#community#benchmark#calibration

Community

jjd-lab/jev-synthetic-surveyPython

S1MB

System One Mosaic Benchmark: a Python evaluator and Next.js viewer comparing Jev and open decision models across 137 English Choice, Noul and Score benchmarks from 106 dataset subsets, with recorded evaluation conditions and a public Hugging Face leaderboard.

Jev Benchmarks & Evaluations#community#benchmark#leaderboard

Community