User repository on GitHub
Speed and accuracy benchmark: TypeSafe's Jev API vs the local Laya MLX typed-decision model on synthetic tasks
- Forks
- 0
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Benchmarks & Evaluations
Jev vs Laya benchmark
Benchmark comparing TypeSafe Jev 1.13 with the local Laya MLX typed-decision model across 1,470 synthetic items and eight tasks: accuracy, calibration, latency and throughput, with scripts, datasets, predictions and the full report in the repo.
- Stars
- 5
- Language
- Python
- License
- MIT
- Reading time
- 3 min
- Published by
- Community
- Author
- harrymunro
- Last verified
- Oct 2026
Used inJev for Software & DevTools
Highlights
- Compares hosted Jev 1.13 with Laya 421M running locally on MLX over 1,470 synthetic items across eight typed-decision tasks, 3,386 judgments per backend.
- Overall correctness: Jev 92.9% against Laya 65.3%; by type Jev scored 99.3 Choice, 98.6 Noul and 76.1 Score against Laya's 68.3, 71.3 and 51.5.
- Latency: single-question p50 136 ms for Jev, about 50 ms of it network, versus 42 ms for Laya; 50 questions in one request took 170 ms against 1,002 ms.
- Throughput: 72 items/s at 32 in flight for Jev against 24 items/s sequential for Laya; Jev won all 19 questions with paired confidence intervals excluding zero.
- Ships generation, speed and accuracy scripts, calibration and bootstrap CIs, per-item predictions and charts; Laya checkpoints download at about 840 MB.
Quickstart
bash
uv venv --python 3.12 .venv && source .venv/bin/activate
git clone https://github.com/mizorewww/laya-mlx && git -C laya-mlx checkout fc1df62828a3fedf4d8229fdac1cbd85f1cdf337
uv pip install ./laya-mlx -r requirements.txt
export TYPESAFE_API_KEY=...
python bench/run_accuracy.pyWatch out
Copy reviewed against the repo README on 2026-10-15. All 1,470 items are synthetic and were generated for this benchmark; gold labels are generation targets filtered by a blind re-label, and the report carries label-noise estimates and a ceiling caveat. This is the authors' own benchmark, not independent confirmation.
Questions about this entry
Short answers drawn from the linked project or page.
Jev 1.13 — TypeSafe's hosted System One model, jev-1.13.0 — against Laya, open-weight typed-decision models run locally with MLX on Apple silicon. The study uses 1,470 synthetic items across eight typed-decision tasks plus controlled latency and throughput sweeps.
Jev, across the board: 92.9% of 3,386 judgments correct against Laya 421M's 65.3%; by question type Jev scored 99.3 Choice, 98.6 Noul and 76.1 Score against 68.3, 71.3 and 51.5. Jev won all 19 questions with paired confidence intervals excluding zero.
It depends on the workload. On a single short question Laya is about 3x faster: 42 ms versus Jev's 136 ms p50, about 50 ms of which is network. With 50 questions on one state in one request Jev answered in 170 ms against Laya's 1,002 ms; Jev is also faster on states past Laya's 512-token window.
You need an Apple silicon Mac for Laya, a TypeSafe API key and — only to regenerate data — an OpenAI key. The README's steps: create a Python 3.12 venv with uv, clone laya-mlx and check out the pinned commit, install it with requirements, export TYPESAFE_API_KEY, then run the bench scripts.
All synthetic, generated for this benchmark. Gold labels are the generation targets filtered by a blind re-label. The moderation and guard sets deliberately contain toxic posts, threats with slurs masked and one-line harmful or jailbreak-style requests; the report gives label-noise estimates and a ceiling caveat.
MIT for the code, data and results in the repository. Laya's weights and laya-mlx are Apache-2.0 and belong to their authors, and Jev is a hosted service under TypeSafe's terms.
It is one team's benchmark on synthetic development data, not independent confirmation, and the README points to calibration analysis, long-context and multilingual results and qualitative observations in the full report. Only 421M Laya checkpoints were tested; per-task breakdowns matter more than the headline.
More Jev resources like this
Independent measurements of typed-decision models: Jev (TypeSafe API) against Laya (open weights) on 40 Chinese support-ticket classifications, plus a local-first cascade that matches Jev's accuracy at about 1.8x the speed.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
yibie/laya-jev-labPython
yibie/laya-jev-labPythonA measured audit of 300 public pages for AI citability using nine Jev decisions per page, checked against an LLM judge: Jev agreed on 85.8% of 900 decisions and cost 158 times less per page, with latency measured for both.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
stas4000/jev-geo-auditPython
stas4000/jev-geo-auditPythonBenchmark harness that scores TypeSafe Jev against any OpenRouter model or local checkpoint on labelled classification data: accuracy, calibration (ECE), confidence distribution, latency and cost across five tasks with 300 items each.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
4esv/jev-evalPython
4esv/jev-evalPythonBenchmark harness comparing Jev, Jeff, Mizan and Laya on 480 authored general Persian questions, 560 literary questions and a 24-excerpt classical poetry pilot, with raw responses saved and answers scored locally without a runtime model judge.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
armanjr/jev-persian-benchmarkPython
armanjr/jev-persian-benchmarkPythonA reproducible 100-task benchmark comparing Jev with Qwen3.5 0.8B, 2B and 4B across three thinking budgets: Jev scored 77.0%, the 2B model at 2,048 tokens 82.0%, and the 4B 96.7%.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
softpudding/jev-frontier-100Python
softpudding/jev-frontier-100PythonA Python research harness evaluating Jev (TypeSafe System One, jev-1.13.0) as a relevance judge for task-level tool spaces. One API call scores every tool with a yes/no noul question per tool, and the repo benchmarks keep-all versus per-request, per-task and trace-based spaces on TGB and BFCL.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
xuan7zhang/jev-toolspacePython
xuan7zhang/jev-toolspacePythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.