Why is replacing the rescoring model in a neuroprosthesis hard?

General language models asked to pick one sentence from a list answer from where a label sits in the list rather than from the sentence itself, so their choices track position rather than content.

How is rescoring posed for Jev?

As a single typed decision, one call that returns a probability for every candidate sentence, combined with the decoder's own score.

What are the word error numbers on the held-out sentences?

On 978 held-out sentences from a participant with ALS, where the published decoder alone reaches 8.1% word error, Jev reaches 7.5% against 7.8% for both OPT-6.7b and Qwen2.5-7B.

What happens when the decoder's weight is re-tuned?

Jev reaches 6.9% against 7.2% and 7.4%, ahead in all four comparisons and at most 0.2 points behind at the 95% bound.

What does Jev cost for speech rescoring?

0.07 USD per thousand sentences, and it needs no GPU; a dedicated GPU running a 7B model is cheaper per sentence only above 43% utilisation.

How fast is the end-to-end rescoring system?

End-to-end latency over the internet is 262 ms, of which 62 ms is spent at the provider, the same order as a 7B model on a local GPU (27 ms) but not faster.

Where are the code and data for this study?

Code and data are available at https://github.com/gabrycina/how-much-language-model.

CommunityJev Benchmarks & Evaluations

Jev Matches 7B Language Models for Speech-Neuroprosthesis Rescoring (arXiv 2609.33538)

A study posing speech-neuroprosthesis rescoring as a single typed Jev decision. On 978 held-out sentences from a participant with ALS, Jev reaches 7.5% word error versus 7.8% for OPT-6.7b and Qwen2.5-7B, at USD 0.07 per thousand sentences with no GPU needed.

See highlights

Open arxiv.org gabrycina/how-much-language-model

Stars: 0
Language: Python
Format: Article
Published by: Community
Last verified: Oct 2026

gabrycina/how-much-language-model

User repository on GitHub

View on GitHub

How much language model does a speech neuroprosthesis need? Measuring the marginal value of second-pass rescoring as the first-pass decoder improves.

Forks: 0
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.

Tagscommunity research arxiv speech calibration cost

Works forJev for Ranking Jev for Scoring Jev for Evaluation

Used inJev for Healthcare & Life Sciences Jev for Software & DevTools

Languages: Python

Highlights

Poses rescoring as one typed call that returns a probability for every candidate sentence.
On 978 held-out sentences from a participant with ALS, the published decoder alone reaches 8.1% word error.
Jev reaches 7.5% against 7.8% for both OPT-6.7b and Qwen2.5-7B; with the decoder weight re-tuned, 6.9% versus 7.2% and 7.4%.
Costs 0.07 USD per thousand sentences and needs no GPU.
End-to-end latency is 262 ms, 62 ms of it at the provider.

Watch out

arXiv preprint posted September 27, 2026 (cs.CL, eess.AS) by Gabriele Cinà; 8 pages, 1 figure, 4 tables. The margins are small (at most 0.2 points behind at the 95% bound in some comparisons), the measurements cover one participant's held-out sentences, and the latency is the same order as, not faster than, a 7B model on a local GPU.

Questions about this entry

Short answers drawn from the linked project or page.

Reactions & coverage

Posts, threads, and videos about this entry from around the web.

X: 100,000 viral posts in 20.4 seconds

X: SEO and GEO fixes, 90% cheaper

X: Hard-coded rules moved to Jev

YouTube: Joe Maddalone tries Jev as a routing and filtering step, with a transcript-analyzer demo.

Related terms

Glossary definitions related to this entry.

More Jev resources like this

TypedBench: A Benchmark for Calibration, Framing Sensitivity, and Cost in System One Decision Models (arXiv 2610.11392)

A benchmark for typed decision models such as Jev, built from seven policy-labelled generators and nine evaluation suites. It reports accuracy as medians across paraphrases, calibration against a matched noise floor, and realized cost under asymmetric cost matrices.