User repository on GitHub
- Forks
- 0
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Benchmarks & Evaluations
Jev Speed Race
A benchmark that races Jev against a frontier chat model on the same job — routing 40 customer-support tickets into four categories — and reports time, tokens and cost per ticket, with an animated replay site and simulated demo results.
- Stars
- 1
- Language
- TypeScript
- Reading time
- 2 min
- Published by
- Community
- Author
- stevenjb5M
- Last verified
- Oct 2026
- Languages
- TypeScript
Highlights
- Same 40 tickets, same four categories, same descriptions: only the answering model differs.
- Jev answers a typed choice question; Claude gets a system prompt and a JSON schema with an enum.
- Price table: Jev at 0.042 USD per 1M input tokens with output free vs Opus 5.5 at 4 and 20 USD.
- npm run race:demo runs fake racers with no API keys; results save to results/ and docs/data.
- An animated replay site with a scoreboard shows results at stevenjb5m.github.io/Jev.
Quickstart
bash
npm install
npm run race:demoWatch out
README and GitHub metadata reviewed 2026-10-17; the digest truncates the fairness notes. The benchmark is the author's own fixture of 40 tickets, the preview results are built from simulated data, and the competitor price cited is Opus-class list pricing.
Questions about this entry
Short answers drawn from the linked project or page.
A benchmark that races Jev against a frontier chat model on the same job: routing 40 customer-support tickets into four categories. It scores right or wrong, time, tokens and dollars for each ticket.
Jev answers a typed choice question through client.systemOne(); Claude gets a system prompt and a JSON schema with an enum of the four teams. The tickets, categories and descriptions are identical for both.
Its comparison table lists Jev at 0.042 USD per 1M input tokens with output free, against 4 USD input and 20 USD output per 1M for Opus 5.5. Those are list prices, not measured invoices.
Node 20 or newer: npm install, then npm run race:demo for a fake race that needs no API keys, or copy .env.example to .env, add the two keys and run npm run race for real results, charts and saved data.
Yes. A GitHub Pages site hosts an animated replay with a scoreboard and every ticket, and results/demo/ contains a preview built with simulated data; running the race updates the site with your latest real data.
Calls run one at a time so the race measures latency rather than throughput, each racer gets one untimed warm-up call, Claude runs at its lowest effort setting, and some tickets are tricky on purpose.
It is the author's own fixture of 40 tickets and four categories, the demo preview uses simulated data, and swapping in a different Claude model means updating its price in the source; it is not an independent evaluation.
More Jev resources like this
A learning lab for decision models: TypeSafe Jev and Microsoft Decision-1 route support tickets in under a second for about $0.02 per 1,000 calls, while Claude handles the hard cases. It ships an interactive site, a benchmark, calibration tests and docs.
Jev Demos & Experiments#community#github#benchmark
A benchmarking studio and real-time simulator that pits Jev (remote TypeSafe System One), a local Laya model and the optimal blackjack Basic Strategy against each other on identical deterministic shoes. It scores every deviation turn by turn and runs headless batches of up to 100,000 rounds.
Jev Benchmarks & Evaluations#community#github#blackjack
Community
mrbanana55/decision_blackjackPython
mrbanana55/decision_blackjackPythonA self-contained playground for evaluating structured judgment requests against LLM providers: validate a SystemOneRequest, run it through an emulator, a JEV baseline via typesafe.ai and an OpenAI-compatible judge, then compare results on three web screens.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
duberblock/jevalsTypeScript
duberblock/jevalsTypeScriptA local Java benchmark that plays 10,000 deterministic transactions through two fraud pipelines: every one to Qwen via Ollama, or through System One/Laya first, where a low-risk answer at confidence 0.70 or higher stops before the chat model.
Jev Benchmarks & Evaluations#community#github#java
Community
AliveMonster/JEVJava
AliveMonster/JEVJavaThird-party TypeScript check that compares Jev (typesafe-ai/jev) against openai/gpt-4o-mini and anthropic/claude-sonnet-4.5 under identical conditions, routing 60 synthetic booking inquiries in four languages for a photo-shoot service, all through one Vercel AI Gateway key.
Jev Benchmarks & Evaluations#community#benchmark#comparison
Community
Shogo-nfrealmusic/jev-evalTypeScript
Shogo-nfrealmusic/jev-evalTypeScriptTypeScript gateway that fans one payload out to seven Jev risk and routing questions in one call, then enforces deterministic policy verdicts.
Jev Benchmarks & Evaluations#community#github
Community
intelliDean/reflexgateTypeScript
intelliDean/reflexgateTypeScriptFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.