Skip to content
JevDirectory.orgTraffic count unavailable. View site traffic
CommunityJev Benchmarks & Evaluations

System One on Snapdragon

A dated harness for running open System One typed-decision models — Laya, Decider, imajev, Winnow-12B, decider-12b — on a Snapdragon X Elite NPU, GPU, and CPU, benchmarked against TypeSafe's Jev API.

See highlightsOpen github.com
Stars
1
Language
Python
License
APACHE-2.0
Published by
Community
Author
esterhuizen
Last verified
Oct 2026
Tagscommunitypythonsnapdragonnpubenchmark
Works forJev for EvaluationJev for Classification
Used inJev for Software & DevToolsJev for Customer Support
Languages
Python
Capabilities
Open sourceRuns locallyRequires API key

Highlights

  • Laya runs on the Hexagon NPU at about 134 ms per request, matching CPU answers on 99.8%, and on the Adreno GPU at about 590 ms.
  • Decider-12b on the NPU reaches JevBench standard 0.986, level with Jev, at about 2.2 s per request in 4-bit.
  • Laya fine-tuned on Claude Opus labels went from 63.5% to 88.5% bucket accuracy on held-out helpdesk tickets, against Jev's 83.5%.
  • Across roughly 3,900 labelled public items the seven-task average is Jev 0.815, Winnow-12B 0.782, decider-12b 0.722, Laya 0.546.
  • Winnow-12B runs entirely on the Hexagon NPU at 4-bit, about 2 s per request, through ONNX Runtime's QNN provider with no test-signing.

Watch out

Apache-2.0, but the README calls it a dated snapshot rather than a maintained fork: versions and hashes are pinned, fixes are proposed upstream, and the tested path is Windows 11 ARM64 on a Snapdragon X Elite with native ARM64 Python, since x64 Python runs emulated.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

Zero-Shot IE Bench
7GitHub stars
A benchmark harness and web UI that cross-compares dozens of zero-shot information-extraction and classification systems — extractor encoders, rerankers, and typed-decision engines like Jev — on accuracy, latency, and cost.
Jev Benchmarks & Evaluations#community#python#benchmark
Communityumstek/zero-shot-ie-benchPython
What the Jev
9GitHub stars
A reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.
Jev Benchmarks & Evaluations#community#python#evals
Communityketa1930/what-the-jevPython
hev reranker
17GitHub stars
A BEIR reranking recipe that asks up to 30 parallel Noul relevance questions in one Jev state, with RESULTS.md against cross-encoders and hosted rerankers.
Jev Benchmarks & Evaluations#community#evaluation#python
Communityhev/rerankerPython
Jev Demos
7GitHub stars
Five self-contained Jev demos, one folder each: Jev fetching for Claude, Jev and Laya grading 500 AI answers, a prompt-injection review gate, and two fine-tuned decider comparisons against Jev.
Jev Demos & Experiments#community#python#demos
Communitymani-aiml/jev-demosPython
jev vs. open alternatives
8GitHub stars
Notebook benchmark that compares Jev with open models and specialised tools on five document-pipeline chores: language detection (Choice(12)), page orientation (Choice(4)), RVL-CDIP classification (Choice(16)), bundle splitting and parse-tier triage, with liteparse doing all PDF rendering and OCR work.
Jev Benchmarks & Evaluations#community#evaluation#document-processing
Communityrun-llama/jev_vs_ossJupyter Notebook
zh-decision-bench
3GitHub stars
The first Chinese-language calibration benchmark for Jev-class System One decision models: 378 items and 467 questions across voice routing, e-commerce service and content moderation, with accuracy, ECE, Brier, NLL, AUROC, option-order and zh-CN/zh-TW robustness metrics.
Jev Benchmarks & Evaluations#community#benchmark#calibration
CommunityCodyQin/zh-decision-benchPython
Back to all resources

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai