iammrduncan/typesafe-ai-benchmark
Organization repository on GitHub
This is a LLM Gateway that mimics typesafe ai structured output. Like an imposter Jev.
- Forks
- 5
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.Homepage
Jev directory
CommunityJev Practices & Patterns
TypeSafe AI Benchmark
Side-by-side benchmark of TypeSafe Jev, Qwen 3.8 27B on Cerebras, and a local Needle 3 across seven synthetic workloads, recording validated outputs, mistakes, latency, tokens, and estimated cost. Raw exports and per-scene limitations are published.
- Stars
- 38
- Language
- TypeScript
- License
- MIT
- Reading time
- 3 min
- Published by
- Community
- Author
- iammrduncan
- Last verified
- Sep 2026
About this project
LLM-native structured output vs. TypeSafe Jev: latency, cost, and judgment quality. How does a fast general-purpose LLM compare with a purpose-built judgment model on the same application tasks? This benchmark runs Qwen 3.8 27B on Cerebras and TypeSafe Jev side by side across seven synthetic workloads, with a separate local Needle 3 evaluation on the same contracts. It records validated outputs, mistakes, request latency, token usage and estimated cost.
Quoted from the project's README. Read the full README.
Works forJev for Evaluation
Used inJev for Software & DevTools
Highlights
- Runs Jev, Qwen 3.8 27B on Cerebras, and a local Needle 3 side by side across seven synthetic workloads.
- Jev logged 176 ms p50 and $0.011919 estimated cost against Qwen's 215 ms and $0.310581.
- All three approaches validate outputs before applying simulated actions.
- Needle 3 was measured separately on an Apple M4 Pro at 864 tok/s median native decode.
- Raw exports, source hashes, and per-scene limitations are published under docs/benchmarks.
Quickstart
bash
test -f .env || cp .env.example .env
npm run setup:needle # optional local Needle 3 lane
npm run check # type checks, lint, offline tests, both buildsWatch out
MIT-licensed. Requires Node 22 and npm 10, API keys for the cloud lanes, and the Hugging Face CLI for the optional local Needle lane; the Needle timing is not a controlled speed ranking.
Questions about this entry
Short answers drawn from the linked project or page.
One synthetic run on one development machine, and the README says these measurements are not calibrated quality or general model parity. The decision API logged 176 ms p50 against 215 ms for Qwen on Cerebras, and $0.011919 against $0.310581 estimated cost, with all seven scenes completing and no rate limits or retries. It is explicitly not a controlled speed ranking.
It is a deliberately narrow baseline and the README says so. Qwen runs with reasoning disabled and a compact output schema, with application fields compiled into numeric slots and one constrained response; it is not asked to emulate decision probabilities. The authors also note that a continuous-batching engine without padding would close most of the speed gap seen on the non-shared-state scenes.
No, and the README separates three different measurements: valid output shape, judgment quality and performance. The agreement table is exact match against local expected outputs, not evidence of calibration or any security property. Repeated synthetic cases are not an independent population sample, invalid outputs count as non-matches, and every mismatch is preserved in the raw exports.
You need Node 22 and npm 10 plus keys for both cloud lanes in your env file. Live runs incur charges and the interactive UI has no cumulative spending cutoff, so use a bounded budget and keep partial or failed runs. A rejected key stops that lane and names the setting to fix, and there are no automatic retries or replacement answers anywhere.
Tickets, routing, driving, guardrails, approvals, scoring and home. Tickets routes 100 customer tickets to team, priority, action and escalation; guardrails runs 100 allow/block checks; approvals makes 100 proposed agent commands with none executed; scoring rates 100 candidate answers against golden references; home runs 24 commands over simulated lights, blinds and temperature.
No. Needle was measured separately the next day, directly through the demo runtime on an Apple M4 Pro, with no browser or HTTP overhead and no competing lane, while Qwen and Jev ran in a browser the previous day. The README calls it not a controlled speed ranking, and its median native decode rate of 864 tokens/s is distinct from its 225 ms median successful request.
No. npm run check runs strict type checks, lint, offline tests and both builds, and normal tests never use provider credentials. Keys are only needed for the live cloud lanes, and the project is MIT licensed. Driving needs WebGPU on localhost or HTTPS, and no real devices or external commands are operated.
More Jev resources like this
A personal-assistant agent with 100 mocked tools that measures how many steps a task takes when the LLM picks the tool versus when Jev picks it before every model step.
Jev Practices & Patterns#community#typescript#agents
Community
vinilana/jev-eval-agentHTML
vinilana/jev-eval-agentHTMLIndependent cross-model benchmark for Jev-class decision models, running 534 frozen cases per complete entrant with scoring code and a four-axis score of accuracy, calibration, latency and cost.
Jev Practices & Patterns#community#python#benchmarks
Community
fstandhartinger/jevbenchPython
fstandhartinger/jevbenchPythonAn experiment comparing Jev with GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 as evaluators of five frozen weather-agent runs, measuring pass-or-fail accuracy against human labels plus variance, cost, and latency.
Jev Practices & Patterns#community#python#evaluation
Community
danielgshea/jev-as-a-judgePython
danielgshea/jev-as-a-judgePythonA probability-aware evaluation harness that compares TypeSafe Jev with GLiNER2.5 on zero-shot single-label text classification, measuring calibration, coverage at a fixed error budget, latency, and token cost.
Jev Practices & Patterns#community#python#benchmarks
Community
AbdelStark/jev-benchmarksPython
AbdelStark/jev-benchmarksPythonAn independent calibration study of Jev against a frontier fallback on Banking77 and Web of Science, packaged as the janus-decide tool that measures a routing threshold on your own data instead of shipping a default one.
Jev Practices & Patterns#community#python#routing
Community
FirasSX914/JanusPython
FirasSX914/JanusPythonA pre-registered, independently rescored audit of Jev's calibration and type safety: calibrated on CLINC150 (ECE 0.0204), systematically overconfident on Banking77 (ECE 0.0936), and zero type errors across 8,576 responses.
Jev Practices & Patterns#community#python#calibration
Community
jourdanlabs/assay-001Python
jourdanlabs/assay-001PythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.