Are the accuracy numbers from real model calls?

Not in the headline experiment. LLMRouterBench ships precomputed scored answers, so the router is evaluated by lookup and no inference happens at all. The RouterArena pilot likewise reads cached answers and only pays for the router's own decision call. That makes both cheap and reproducible, but neither measures a system that actually generates answers.

Does the win come from Jev or from the retrieval evidence?

From the retrieval evidence, by the author's own ablation. Replacing Jev with a dev-set prior lands on essentially the same frontier, 62.4% at marginally lower cost at the cost-sensitive end, so the difficulty signal was largely redundant with neighbour evidence here. The README reports that rather than crediting Jev for the gain.

Is this comparable to the public RouterArena leaderboard?

The pilot is not: 100 examples from a subset, over the three cheap models that ship with cached answers. The full-run harness is in place and its scorer reproduces RouterArena's script to four decimals, but that run has not been executed. Estimated cost for it is $35 to $70, almost all of it selected-model answers since only three models have cached results.

What does it actually cost to run these routers?

Jev's own spend is negligible: roughly $0.13 across 5,835 difficulty calls for the large experiment and under $0.006 for the pilot, consistent with being billed on input tokens with output free. The expensive part is candidate-model inference, which is why the unrun full pass is budgeted in the tens of dollars while Jev routing stays under a dollar.

What was the headline result on LLMRouterBench?

On the 13-model flagship pool, over 11,668 queries split into dev and a held-out eval half, a Jev-difficulty plus retrieval router beat the best single model: 62.4% versus 60.3% accuracy at lower cost, or best-single accuracy at 33% lower cost. Evaluation used lookup on precomputed answers, so there was no inference cost.

What did the pilot on RouterArena's cached models show?

On a 100-example subset, Jev with a difficulty rubric reached 69.5% routed accuracy at a median 311 ms decision latency, edging out GLiClass at 69.1% and 1.46 s. No router beat the best single model there, because gemini-2.0-flash-001 was both the cheapest and the most accurate model on that small pool.

What is the frozen difficulty rubric?

It is model-agnostic and not derived from RouterArena's own difficulty labels: easy is a single fact recall or one-step lookup, medium is two to four dependent reasoning steps in one domain, and hard is long multi-step reasoning, specialized knowledge, or code that must run.

CommunityJev Practices & Patterns

Jev Routing Experiment

Runs Jev as an LLM router over two public benchmarks, with a no-Jev ablation and RouterArena's own scorer so the numbers sit next to the public leaderboard.

See highlights

Open github.com

Stars: 2
Language: Python
License: APACHE-2.0
Published by: Community
Author: TokenTrim
Last verified: Oct 2026

TokenTrim/jev-routing-experiment

Organization repository on GitHub

View on GitHub

Benchmarking TypeSafe's Jev decision model as a cost-efficient LLM router on RouterArena

Forks: 2
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

Can TypeSafe's Jev decision model route queries across a pool of LLMs cost-efficiently? This repo runs Jev as a router on RouterArena (ICLR 2026, arXiv:2510.00202) and scores it with RouterArena's own offline scorer, so the numbers are directly comparable to the public leaderboard.

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import awesomejev

Works forJev for Classification Jev for Routing Jev for Evaluation

Used inJev for Software & DevTools

Languages: Python

Highlights

On a 13-model pool, a Jev-difficulty plus retrieval router reaches 62.4% against the best single model's 60.3%, or matches that accuracy at 33% lower cost.
The ablation is the interesting part: dropping Jev lands on the same frontier, so the author credits retrieval evidence, not the difficulty signal.
A separate RouterArena pilot over 100 cached queries reports 69.5% at a median 311 ms decision, next to GLiClass at 69.1% and 1.46 s.
Scoring reuses RouterArena's rule-based metrics with no LLM judge, and the repo's Arena score reproduces the official script to four decimals.

Watch out

The two experiments are separate and should not be read as one result. The large LLMRouterBench number is a lookup over precomputed answers, so no model is ever called and it says nothing about inference-time behaviour, while the RouterArena pilot covers only 100 examples from a subset over the three models that happen to have free cached answers, which the author notes is not comparable to full leaderboard entries. In that pilot no router beat the best single model, because one model was both cheapest and most accurate. GLiClass ran on a free hosted demo whose latency includes network and queue time, and no production cost claims are made from it. A large oracle gap remains between the upper bound and the router. The full 8,400-query RouterArena run has not been made, and is estimated at $35 to $70, mostly for selected-model answers. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-on-a-laptop

Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.