Jev directory
CommunityJev Benchmarks & Evaluations
Decision Index Results: Kieve-JeV Fractal 300M
Hugging Face dataset of run results for the kieve-jev-fractal-300m decision engine, recording per-benchmark tallies of requests, answered, abstained and errored cases along with scores, latency medians and p95s, and retrieval and cost-aware metrics.
- Reading time
- 2 min
- Published by
- Community
- Added
- Oct 2026
Highlights
- Benchmark tracks include RouterBench 0-shot and 5-shot, GSM8K 10-choice and 4-choice, and five iSarcasmEval splits.
- Latency fields report median, p95 and mean milliseconds for successful requests.
- Custom metrics include Brier score, log loss, nDCG@10, MRR, recall@10, BM25 nDCG@10 and candidate recall.
- Cost-aware detail carries quality and cost utility, oracle-optimal values and utility regret.
- Model provenance fields record repo, checkpoint, config, device, variant, temperature and policy.
Questions about this entry
Short answers drawn from the linked project or page.
It is a Hugging Face dataset published under the scairesearch organization by ScaiResearch Labs. It holds the structured results of a decision-index evaluation run, with one row set covering engine configuration, benchmark scores and latency statistics.
The results belong to the kieve-jev-fractal-300m decision engine. A model_source struct records the kind, repo, checkpoint, config, device, variant, temperature and policy used for the run, so the exact configuration is traceable.
A successful_request_latency_ms struct reports median, p95 and mean values for successful requests. Individual benchmark entries also carry a median_ms field alongside their score.
Detail fields include Brier score, log loss and accuracy, plus retrieval metrics such as nDCG@10, MRR, recall@10, candidate recall and a BM25 nDCG@10 baseline for comparison.
Yes. Custom metrics include quality_cost_usd, quality_utility, quality_oracle_optimal, quality_utility_regret, cost_aware_cost_usd, cost_aware_utility, cost_aware_oracle_optimal, cost_aware_utility_regret and value_regret.
No. RouterBench and GSM8K tracks carry numeric scores, while iSarcasmEval tracks for A-Ar, A-En, B-En, C-Ar and C-En show null scores with their scored request counts still recorded.
Each benchmark row lists requests, answered, unsupported, errors, abstained, pending and scored_requests counts, plus catalog_id, dataset name, metric name and score.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Hacker News: ASTGL replay chose local Laya routing over hosted Jev on one Mac Studio setup.
Jmeg8r15 pointsDoes an open-weight decision model beat a hosted one? Jev vs. LayaRead the thread on Hacker NewsReddit: A walkthrough of routing between models with Jev.
X: Standard One SH (StandardThinking)
Facebook: Intake Router keeps diagnosis out of the answer space at the front desk.
FacebookClinicnotes
More Jev resources like this
Benchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
Controlled benchmark of Jev against Claude Sonnet 5 on one coding-agent decision: given a bug report, which file needs editing? It scores 200 real SWE-bench Verified issues with cost, latency, and confidence-band routing tables.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
CommunityK624/jev-localization-bench
Hugging Face dataset of synthetic typed-decision records for Jev-style control and probability-estimation tasks. It holds 12 subsets with question, options, target, state_json, and metadata_json fields across train, calibration, validation, test, and ood splits.
Jev Benchmarks & Evaluations#open-models#calibration#evaluation
Community
An evaluation results dataset publishing per-benchmark outcomes for scairesearch/kieve-jev-moe-300m, including request, answered, abstained, and error counts, scored metrics, successful-request latency medians and p95s, and the runner configuration used for the run.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
Community
Spring Boot and Spring AI benchmark comparing Java keyword rules, Jev 1.13.0, and Claude Haiku 5.5 on warehouse goods-receipt triage, with a 200-remark synthetic dataset and a Python analysis script.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
Communitymohammedahmadi1990/jev-triage
arXiv study applying Jev and Clef-Flash to manufacturing defect detection with as few as two reference images per class, including agentic refinement of text descriptions and probability calibration. Ships replication code, an R decisions package, and a no-code web app.
Jev Benchmarks & EvaluationsArticle#arxiv#calibration#classification
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.