Jev Benchmarks

A ranked comparison of thirteen decision-model APIs as of October 2026, including Jev, OpenAI Decisions, Clef, Liquid d1, Kev, Perplexity Decider, and Strands, scored on evidence, usage, and price.

GIGAZINE's report on Liquid AI's d1, which it describes as the first decision model to beat Jev on Hugging Face's Decision Index, and a short survey of the clone wave that followed Jev's September launch.

The 400-case, 2,000-decision benchmark open models train and report against: four workflows, state plus typed questions, and gold distributions averaged from three teacher samples.

An independent 2,018-item answer-verifier benchmark: the same items and labels for every entrant, each run on its own maker's code, with Jev, ZTC-Judge, Darwin, and open models ranked by weighted AUC.

The System One Mosaic Benchmark leaderboard: about 100 decision models compared across 100+ specialized tasks, with results regenerated from a public dataset rather than estimated.

TypeSafe's 11 September 2026 essay against benchmaxxing: a public eval can select the model even when nobody trains on that test.

An unofficial Hugging Face space that tracks open Jev reproductions and scores the ones that finished a frozen benchmark suite.

The index of TypeSafe's worked examples, from parallel questions and reranking to guardrails, date extraction, and self-consistency, each with datasets and measured results.

Scott Chacon runs hosted Jev against two local decision models, laya and kev, in a Tetris match and a GitHub Settings filter, with a public test repo.

A head-to-head benchmark that prices 200 products from descriptions with Jev against GPT-5.6 Luna and GPT-4.1-Nano.

A release guide with eval tables, pricing, curl and Python examples, and fit boundaries for Jev's early access.

A heavily footnoted claims audit that separates documented Jev facts from marketing, including the seed round, third-party tests, and missing transparency.

Published evaluations of four automation workflows, security incidents, agent trace observability, invoice processing, and customer service, comparing Jev and frontier LLMs as structured workflows versus single prompts.

Side-by-side benchmark of TypeSafe Jev, Qwen 3.8 27B on Cerebras, and a local Needle 3 across seven synthetic workloads, recording validated outputs, mistakes, latency, tokens, and estimated cost. Raw exports and per-scene limitations are published.

Reranking benchmark that gave Jev, Cohere Rerank 4 Pro, ZeroEntropy zerank-2, and DeepSeek the same thirty BM25 candidates across eight English datasets, publishing saved responses, scoring code, and paired-bootstrap intervals. Jev's rubric scored 0.692 nDCG@10 against Cohere Pro's 0.691.

A personal-assistant agent with 100 mocked tools that measures how many steps a task takes when the LLM picks the tool versus when Jev picks it before every model step.

Blind security benchmarks for Jev on public corpora: 662 prompt-injection messages and 200 matched vulnerable-code pairs, with raw per-sample results and a TUI dashboard.

An exploratory study of zero-shot ham, spam, and phishing classification with Jev: enriched email context raised main-set accuracy from 93.62% to 98.64%, within 0.23 points of a trained TF-IDF baseline.

An open 0.6B replica of Jev that turns states and questions into full probability distributions without decoding answer tokens, trained and evaluated on Maze, Snake, and ViZDoom.

An independent open reproduction of the System One model class: Qwen3.5-based 2B and 35B mixture-of-experts models that return typed Choice, Score, and Noul probabilities in one forward pass, with nothing distilled from Jev.

An independent baseline that reads typed option probabilities straight from a frozen Qwen3.5-4B's logits in one forward pass, reproducing Jev's interface pattern with open models rather than Jev's undisclosed model or training.

A benchmark that runs Jev on all 6,257 text traces of Who&When Pro to attribute agent failures, scoring it with the official pinned scorer against the paper's GPT-5.4, Claude Sonnet 4.6, GLM-5, and Qwen3.5-122B baselines.

Apache-licensed, locally runnable Jev-style decision models (0.8B, 4B, 9B on Qwen3.5) with released weights, training code, a System One-compatible server, frozen eval suites, and a playground.

A graded relevance evaluation of Jev as a reranker: 9,831 labelled pairs from 164 queries over a 33,047-item skills catalog, comparing Jev score reranks with BM25, bge-m3, and rank fusion.

An independent calibration study of Jev over three public benchmarks and 900 rule-generated support tickets, publishing every raw Gateway response and the quantization limits of returned probabilities.

A public benchmark comparing Jev with Claude Haiku 4.5 on whether an email agent should click a link, over 2,000 PhishNChips emails, with calibration, latency, cost, and five signal questions.

A measurement harness that sends each file snippet to Jev as one Noul question asking whether the content contains a usable secret credential, then scores the returned probability against the expected label.

An Apache-licensed Python library that reads option-token probabilities from open-weight models through Transformers or vLLM, giving Jev-style typed decisions in one forward pass without an HTTP server.

A probability-aware evaluation harness that compares TypeSafe Jev with GLiNER2.5 on zero-shot single-label text classification, measuring calibration, coverage at a fixed error budget, latency, and token cost.

An independent calibration study of Jev against a frontier fallback on Banking77 and Web of Science, packaged as the janus-decide tool that measures a routing threshold on your own data instead of shipping a default one.

A reproducible harness that scores Jev Ultrafast research-browser runs over 11 baseline cases plus human and quant stress suites with CoS-locked QC grades, regenerating the field note and trace notebooks offline.

A Next.js playground that benchmarks Jev against GPT-5.6 Luna, Claude Haiku 4.5, Gemini 3.5 Flash Lite, and GPT-6 Astra in tic-tac-toe and Connect Four, giving every model the same legal-move Choice options through the Vercel AI Gateway.

A ModernBERT-based open decision model that evaluates Choice, Score, and Noul schemas in one non-autoregressive pass; Verdict 2.0 reports 77.10% accuracy on 2,000 held-out enterprise decisions.

Unofficial companion to dspy-typesafeify that records and replays TypeSafe calls to measure calibration, selective risk, confidence-gated abstention, latency, tokens and modeled cost for Jev decisions.

Independent measurement of whether ORDER BY over a Jev probability is defensible, pre-registering gates on pairwise inversion, Score ordinality, calibration, and negation and paraphrase invariants.

Independent cross-model benchmark for Jev-class decision models, running 534 frozen cases per complete entrant with scoring code and a four-axis score of accuracy, calibration, latency and cost.

A pre-registered, independently rescored audit of Jev's calibration and type safety: calibrated on CLINC150 (ECE 0.0204), systematically overconfident on Banking77 (ECE 0.0936), and zero type errors across 8,576 responses.

A from-scratch one-pass option scorer that maps a context and a changing list of text options to one probability per option, adding a hashed n-gram encoder, rival-aware attention, a gated head, and temperature scaling; JSONL data is drop-in compatible with jevlike.

Two-stage Jev control of a Franka Panda in MuJoCo: Jev selects a task intent, then X/Y/Z directions and a gripper command executed against real contacts, with 43 of 50 Jev episodes succeeding versus 48 of 50 for a rule baseline.

An encoder-side reproduction of Jev built on a retrained GLiClass ModernBERT base (151M): it evaluates multiple typed questions in one non-autoregressive forward pass, returning choices, ordinal scores, and probabilities under 35 ms.

An open re-creation of Jev's decision model on an open-weights Qwen3.5-4B: it answers a list of typed questions in one forward pass over POST /v1/systemone in about 200 ms, never writing free text.

An experiment comparing Jev with GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 as evaluators of five frozen weather-agent runs, measuring pass-or-fail accuracy against human labels plus variance, cost, and latency.

A comparison arena that runs the same batch of review comments through Jev and DeepSeek, showing processing time, cost, and per-label results with CSV/Excel import, replay, and offline reports.

A self-hosted implementation of TypeSafe's Jev System One API powered by the 400M-parameter GLiFormer encoder: it serves choice, score, and noul and drops into the official typesafe-sdk via TYPESAFE_BASE_URL, but trails Jev on reasoning-heavy tasks.

An independent CUDA and vLLM take on parallel typed decisions: it scores verified single-token labels with the model's own head and assembles JSON in code.

A browsable atlas of 1,018 AI papers from August 2025 to August 2026, sorted into 24 topic collections with labs, citations, and code links, plus a reproducible cost benchmark for summarizing 1,000 papers.

Every's head of evals ran Jev across 27 of his articles plus 10 AI-styled fakes, asking 21 AI-tell questions about each and reporting what the judgments caught and where he would not trust them yet.

An independent measurement spanning 5,477 test rows and 34.1M input tokens, comparing one direct Jev question per row with 12-14 scored dimensions fitted to local labels.

Sixteen thousand calls against gpt-5.4-mini and gpt-5.6-luna on four public datasets plus production pipeline decisions, with a threshold procedure and an explicit filter-not-replacement verdict.

A use-case study comparing Jev, Mistral Small 4, and Gemini 3.5 Flash-Lite on rejecting unsuitable local-event listings, with individually tuned prompts, accuracy, response times, and costs.

An independent benchmark of hosted Jev against six LLMs on PubMedQA, Banking77, and HelpSteer2, scoring decisions against human labels and reporting per-board results for each model.

A measured search reranking run over 33,047 catalog entries, 164 real queries, and 9,831 labelled pairs, reporting how Jev compares with BM25 and bge-m3.

Diogo Almeida's launch thread: after co-inventing ChatGPT, he spent two years on RLCD and Jev, a System One model he pitches as 20-200x faster and 40-400x cheaper for typed decisions.

Yuchen Jin's take on Jev for LLM-as-a-judge work: 20-200x faster and 40-400x cheaper, with a screenshot of the model picking which candidate is AGI after a prompt sweep under $0.10.

The launch story: why TypeSafe built decision models, the side-by-side demo, the Doom and Wikiracing bots, and the published workflow evals.