Jev directory
CommunityJev Benchmarks & Evaluations
Kieve-JeV MoE 300M Decision Index Results
An evaluation results dataset publishing per-benchmark outcomes for scairesearch/kieve-jev-moe-300m, including request, answered, abstained, and error counts, scored metrics, successful-request latency medians and p95s, and the runner configuration used for the run.
- Reading time
- 2 min
- Published by
- Community
- Added
- Oct 2026
Highlights
- Records per-benchmark request, answered, unsupported, error, abstained, and pending counts.
- Includes successful-request latency medians, p95s, and means for the scored run.
- Tracks RouterBench-0shot, RouterBench-5shot, GSM8K-10choice, and GSM8K-4choice scores.
- Carries custom metrics including ndcg_at_10, mrr, recall_at_10, brier, and log_loss.
- Model source fields capture repo, checkpoint, config, device, variant, temperature, and policy.
Questions about this entry
Short answers drawn from the linked project or page.
It is a Hugging Face dataset published by ScaiResearch Labs that holds the results of a decision-index evaluation run for the kieve-jev-moe-300m model, including per-benchmark counts, metrics, latency figures, and the configuration used by the runner.
The tracks recorded include RouterBench-0shot, RouterBench-5shot, GSM8K-10choice, GSM8K-4choice, and several iSarcasmEval variants (A-Ar, A-En, B-En, C-Ar, C-En), each with a metric, score, and scored_requests field.
Yes. It stores successful_request_latency_ms with median, p95, and mean values, and each benchmark entry also carries a median_ms field, alongside environment fields such as device, cuda, gpu, torch, and loaded_seconds.
The schema includes a model_source struct with kind, repo, revision, checkpoint, config, device, variant, temperature, and policy, plus runner_sha256, rows_path, edition, and note fields describing the run.
In the published structure those tracks appear with score set to null while scored_requests is still recorded, so they carry no numeric score value in this file even though the request count is present.
It reports results for scairesearch/kieve-jev-moe-300m, a decision engine of the jev variant that returns calibrated distributions over supplied typed options, so the dataset is a record of how that engine performed on catalogued benchmark tasks.
The dataset is hosted on Hugging Face under the scairesearch namespace with a dataset card and a Files and versions tab where the stored data can be viewed and downloaded.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Standard One SH (StandardThinking)
LinkedIn: What multimodal decision models actually change, and what they do not.
LinkedInJevDirectory.org
Hacker News: Decision Index 0.2.1 scores open reproductions of Jev on a frozen suite.
jackbravo4 pointsDecision Index 0.2.1 – 70 open reproductions of TypeSafe Jev's Decision ModelRead the thread on Hacker NewsHacker News: Pointer to a DiffusionGemma plus vLLM tutorial for a Jev-shaped classifier.
1 pointHow to build a Jev-style classifier with DiffusionGemma and vLLMRead the thread on Hacker NewsMore Jev resources like this
Benchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
Hugging Face dataset of synthetic typed-decision records for Jev-style control and probability-estimation tasks. It holds 12 subsets with question, options, target, state_json, and metadata_json fields across train, calibration, validation, test, and ood splits.
Jev Benchmarks & Evaluations#open-models#calibration#evaluation
Community
Hugging Face dataset of run results for the kieve-jev-fractal-300m decision engine, recording per-benchmark tallies of requests, answered, abstained and errored cases along with scores, latency medians and p95s, and retrieval and cost-aware metrics.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
Community
Controlled benchmark of Jev against Claude Sonnet 5 on one coding-agent decision: given a bug report, which file needs editing? It scores 200 real SWE-bench Verified issues with cost, latency, and confidence-band routing tables.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
CommunityK624/jev-localization-bench
Spring Boot and Spring AI benchmark comparing Java keyword rules, Jev 1.13.0, and Claude Haiku 5.5 on warehouse goods-receipt triage, with a 200-remark synthetic dataset and a Python analysis script.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
Communitymohammedahmadi1990/jev-triage
arXiv study applying Jev and Clef-Flash to manufacturing defect detection with as few as two reference images per class, including agentic refinement of text descriptions and probability calibration. Ships replication code, an R decisions package, and a no-code web app.
Jev Benchmarks & EvaluationsArticle#arxiv#calibration#classification
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.