Jev directory
Jev for Evaluation
660 curated Jev resources. Measure model behavior: evals, calibration, consistency, and observability.
By format
660- Docs61
- Repos & sites404
- Videos12
- Posts28
- Articles145
- Playgrounds10
Updated
All 660 Jev AI model resources for Evaluation
Search the full list, or filter by category, format, and tag.
A podcast episode on Jev as a decision model rather than a chat model: hosts Luis Calderon and Carina discuss roughly 20x faster Coach testing, why that preliminary figure differs from TypeSafe's vendor benchmarks, and why type safety does not guarantee a correct decision.
Jev Guides & ArticlesArticle#community#press#podcast
Community
Braintrust's provider page covers two ways to use TypeSafe Jev: instrumenting System One calls as decision spans, and running Jev as an LLM judge that turns typed choices into scores or labels.
Jev Guides & ArticlesArticle#community#press#explainer
Community
Langfuse's guide to decision-model evaluators runs OpenAI's gpt-6-luna or TypeSafe Jev over observations, where each Choice, Score or Yes/no question becomes one Langfuse score.
Jev Guides & ArticlesArticle#community#press#explainer
Community
Langfuse's notebook integration traces TypeSafe Jev System One calls through the OpenInference instrumentor, which patches the Python client in place and emits OpenTelemetry spans.
Jev Guides & ArticlesArticle#community#press#explainer
Community
Langfuse's September 18 post presents Jev as a decision model for routing, classification and eval verdicts, and walks through scoring production traces with Jev-as-a-judge in Langfuse.
Jev Media & TalksArticle#community#press#explainer
Community
A community roundup of every public Jev evaluation: TypeSafe's own workflow evals plus at least twenty independent builder benchmarks. The site reports speed medians of 127-422 ms per call leading every LLM compared, while accuracy splits by task.
Jev Media & TalksArticle#community#press#calibration
Community
Blackwell Systems' adversarial technical due diligence on Jev, built only from public material and live API calls. The author reproduces TypeSafe's published addressee numbers within a point and finds a real but noise-type-specific robustness moat.
Jev Guides & ArticlesArticle#community#press#competition
Community
An independent hands-on evaluation that ran Jev live on 16,379 benchmark requests and reports it is no frontier model but genuinely useful for cheap real-time sub-frontier judgement, with 82.7% MMLU-Pro and 76.5% GPQA Diamond.
Jev Guides & ArticlesArticle#community#press#competition
Community
A playground operator's audit of every published Jev number: TypeSafe's four-workflow evals, the favourable homepage demo, independent tests and its own calls. It concludes the price and speed are real and the intelligence claim is roughly mid-tier frontier on decision workflows.
Jev Media & TalksArticle#community#press#explainer
Community
Interactive comparison of 21 decision models, with direct-provider and OpenRouter routes measured separately on 1,100 public jev-bench items. The charts plot accuracy against benchmark cost, official prices and median latency, with Jev 1.13 and 1.14 measured separately.
Jev Guides & ArticlesArticle#community#press#calibration
Community
Doug Does' benchmark dashboard for JevBench: 534 questions, 48 ranked models, composite scores that already include cost, head-to-head comparison, latency and a Pareto frontier. Jev 1.13.0 tops the table at 74.40.
Jev Media & TalksArticle#community#press#competition
Community
The desk's index of every Jev measurement it has published, keeping each figure's original caveat: JevBench boards, Vercel gateway reach, Celeris and Artificial Analysis results, and the desk's own Jev Day probes which it says it did not rerun.
Jev Guides & ArticlesArticle#community#press#calibration
Community
Side-by-side tables placing similar Jev tests next to each other, from the JevBench API board to Vercel paid-team reach, with the desk's warning that differing sets, labels and comparators mean the rows are not a ranking.
Jev Guides & ArticlesArticle#community#press#competition
Community
Torchcast AI's 4B text decision model on Qwen3.5-4B returns yes/no probabilities, choice distributions and ordinal score distributions through a TypeSafe-compatible API, reading option letters in one forward pass under CC BY-NC 4.0.
Jev Models & Reimplementations#community#open-models#typed-decisions
Community
Sammyliu's audio decision corpus for reinforcement learning and evaluation: causal audio windows paired with runtime action questions, each receiving a categorical label distribution. Train holds 168,494 decisions over 71,952 source windows; test, dev, calibration and an external diagnostic add about 122,600 more.
Jev Benchmarks & EvaluationsDocs#community#evaluation#audio
Community
gyanbhambhani's source package for running the oss-jev training job on Hugging Face Jobs without a custom Docker image: jobs_uv_entry.py is a PEP 723 entry point, and it downloads the oss-jev-distill pack in-script when /data is not mounted. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#fine-tuning
Community
Teacher-labelled distillation pack for the one-token Qwen decision track: 10,044 rows across train, eval and calibration JSONL files, plus 1,708 labelled fixtures. Labels come from jev-1.13.0, and the card says not to train on the eval or calib splits. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#distillation
Community
The benchmark behind JevAgentHarness's paper on decision models that share control with LLMs: next-tool decisions scored at isolated, open-loop and closed-loop levels, with 3,939 common-state questions and 153,621 answers from 39 models. The L1 question texts are third-party data and are not redistributed. CC BY 4.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#agents
Community
Run outputs behind the JevGPT paper's tables: 830 BrowseComp-Plus records at 90.48% accuracy and 105 NeuCLIRBench topics at 0.5390 nDCG@20, produced by a GPT-5.5 System Two paired with Jev-1.13.0 as System One. Per-item cost, latency and encrypted trajectory files are included. CC BY 4.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#agents
Community
The research record of the Xyntetik typed-decisions track: TD-000 measurements and the preregistered TD-001 and TD-002 tests of whether /v1/decide probabilities from Xyntetik-Kvist-14B are usable as decisions. Both tests are INCONCLUSIVE; the record holds questions, scored rows, analyses and the harness. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
ML Intern Lab's benchmark write-up of open decision models on the LocalLLaMA/typed-decisions test split: 2,000 decisions per model for GEV-26B-Decide, JEV-35B and pplx-decider-v1.1-27b, with accuracy, KL, Brier, ECE and latency tables. The repository states no licence.
Jev Benchmarks & EvaluationsDocs#community#evaluation#benchmark
Community
xiq's JEV-format view of OpenJev-Data: every browser step re-expressed as state, questions and gold across eight configs covering Mind2Web, NNetNav, WebChain, two teacher runs and on-policy GLM, plus a frozen 2,346-step bench and 153 live tasks. Licences are mixed and per-row terms apply.
Jev Benchmarks & EvaluationsDocs#community#evaluation#browser-agent
Community
An evaluation set from the social-jev account: 2,597 validation and 2,553 test rows carrying dataset, document and state ids, a query type, a role/content message list and an integer target. The repository carries no prose card, so the task, label meaning and licence are undocumented.
Jev Benchmarks & EvaluationsDocs#community#evaluation#social-science
Community
johannhartmann's German translation of the LocalLLaMA/typed-decisions benchmark at revision c806f6c: states, instructions, options and rubrics translated to German while keys, ids and gold distributions stay as in the original. Machine-translated without human review. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#typed-decisions
Community
Twelve frozen typed-decision configs from the independent Open-Jev project, publishing controlled and mostly synthetic tasks with probability or ordinal targets: from the 113,568-row default release to the 160,797-row browser and drone expansion. CC0 1.0; English, Chinese and Turkish.
Jev Benchmarks & EvaluationsDocs#community#evaluation#typed-decisions
Community
Elfsong's public Decision Index 0.3 runs for two encoders: JEV Ettin 400M scores 14.11 and a FLAN-T5 encoder 342M scores 18.61, each across 140,178 scoreable requests with zero runtime errors under the official runner and scorers. The repository declares no licence.
Jev Benchmarks & EvaluationsDocs#community#evaluation#benchmark
Community
A multiple-choice visual QA dataset for candidate-dependent evidence, published under the tudo-vn account: each item has one image, one question and two candidate sets of 3 to 10 options, each set needing a different evidence region. For research and development only; annotations are CC BY 4.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#vision
Community
cnfeichen's reproducible Chinese reranking benchmark for Jev over hard negatives from C-MTEB/T2Reranking: a 4-level Jev Score rubric reaches 0.546 nDCG@5 on 100 queries against 0.413 for BM25 with jieba. Independent community research, not affiliated with TypeSafe. MIT.
Jev Benchmarks & EvaluationsDocs#community#evaluation#reranking
Community
The training and validation data behind jaeyong2's clef-flash-jev-claude-distill fine-tune: 37,932 states and 150,827 typed questions with soft target distributions, plus a 19,025-record language pool. The card reports the fine-tune lifts the Decision Index public suite from 52.55 to 54.96. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#distillation
Community
Twelve frozen, mostly synthetic typed-decision configs from the independent Open-Jev project, published as redistributable parquet splits with manifests, raw records and reconstruction instructions; the card states these are not official TypeSafe training data.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
A social-jev dataset for BASIL: the labelled config pairs query types, polarity and messages with integer targets across 201 train, 24 validation and 25 test rows, while the original config keeps 300 span-annotated source rows.
Jev Benchmarks & EvaluationsDocs#community#evaluation#classification
Community
A social-jev dataset for MFRC: labelled rows pair query types, polarity, messages and targets across 312 train, 39 validation and 39 test examples, and the original config keeps 17,844 rows with subreddit and vote metadata.
Jev Benchmarks & EvaluationsDocs#community#evaluation#classification
Community
A social-jev dataset for BABE: labelled rows pair query types, polarity, messages and integer targets across 362 train, 39 validation and 44 test examples, with an original config of 3,709 span-annotated rows.
Jev Benchmarks & EvaluationsDocs#community#evaluation#classification
Community
A CV/job fit-scoring benchmark from gw0: 250 synthetic CVs, 404 job postings and 5,000 pairs, each judged on 17 typed questions with a score and a confidence, built to train and evaluate the Jev-like jobfit-model.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
Metamorphic coherence benchmark for typed probabilistic decision models: 240 synthetic cases and 2,240 questions checked by 50 probability laws, with released reports for 21 open models and 4 reference models.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
Evaluation artifacts for hotchpotch's Bekko System One v0 models on Decision Index 0.2.1: local strict, no-truncation runs scoring 7.37 (17M), 12.83 (68M) and 21.87 (400M), with predictions and environment hashes.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
m-lagnajit's typed-decision dataset v2.1: 21,591 short-text rows with a human gold label, a calibrated soft label and the raw teacher ingredients, plus v2.1's corrected label that swaps the teacher's wrong top picks.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
A benchmark of general decision models that unifies 36 datasets, 58 subtasks and 11,257 examples in one test split, with choice, noul and score questions in English and Chinese and 25 configurations scored.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
A 30-option CWE weakness-class benchmark for Jev-class decision models: pick the class of a real NVD vulnerability description, with a 150-case dev split scored locally and 1,500 held-out test cases on the leaderboard.
Jev Benchmarks & EvaluationsDocs#community#evaluation#security
Community
The public track of a multimodal agentic-judge benchmark for Jev-style models: 3,208 items from 16 benchmarks across five judge roles, scored on option probabilities rather than a top answer, under evaluation-only terms.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
RANIntent v1: an O-RAN intent-interpretation benchmark from UTS with 16 telemetry conditions, where an interpreter returns one A1 policy with 9 fields; 300 test and 60 dev cases per condition.
Jev Benchmarks & EvaluationsDocs#community#evaluation#telecom
Community
English, Turkish and German decision-training rows from the JevAlt project: 26,043 situations with typed questions and probability targets, used to train Deem-4B, Karar-4B and Wähler-4B; Karar-4B reports 96.8% on Turkish.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
hotchpotch's training dataset for the Bekko System One v0 decision family: a 1M–10M-row parquet collection whose configs are named for the public datasets behind it, tagged typed-decisions, choice, noul and score.
Jev Benchmarks & EvaluationsDocs#community#evaluation#typed-decisions
Community
A single paste-in prompt for a coding agent that investigates what Jev, TypeSafe's structured decision model, could make possible in an existing project, from replacing parse-the-text calls to using semantic judgments as routine operations.
Jev Cookbooks#community#github#prompt
Community
ryana/jevify
ryana/jevifyA timestamped benchmark and trading arena that replays resolved prediction markets: models commit to probabilities using only information available at the time, and realized P&L is the headline metric with Brier, log-loss and ECE as checks.
Jev Benchmarks & Evaluations#community#github#python
Community
ruyianry/JevGymPython
ruyianry/JevGymPythonA local-first tool from Neatlogs for watching what your AI agent did: it ingests the traces your agent already sends, checks every run, and gives coding agents 23 MCP tools to read, explain, search, compare, and replay runs.
Jev Tools & CLIs#community#github#typescript
Community
neatlogs/neatrootsTypeScript
neatlogs/neatrootsTypeScriptOne Python interface for evaluating what LLM apps and agents say and do. Its CascadeEvaluator asks a decision model first — Jev, OpenAI's Decisions API, or any model served at /v1/systemone — keeps confident answers, and sends only the rest to an LLM judge.
Jev Benchmarks & Evaluations#community#github#python
Community
grounded-ai/grounded_aiPython
grounded-ai/grounded_aiPythonA CLI from Pinecone's DevRel team for testing Agent Skills and docs against tasks: run tasks across Claude, Copilot and Gemini in local or Modal sandboxes, compare with-skill, without-skill and with-docs variants, and grade with Claude or a TypeSafe System One backend.
Jev Benchmarks & Evaluations#community#github#python
Community
pinecone-io/cultivarPython
pinecone-io/cultivarPythonStart here with the Jev AI model
Featured entries for evaluation.
A desk recording news, benchmarks, and listed API prices for TypeSafe's Jev and other decision models, with each figure linked to the page that printed it. Updated October 15, 2026, and not affiliated with TypeSafe.
Jev Guides & ArticlesDocs#community#news#benchmarks
Community
An open Claude Cowork-style workspace whose eval testkit can call Jev as a typed verification judge on agent-produced artifacts.
Jev Benchmarks & Evaluations#community#typescript#agents
Community
different-ai/openworkTypeScript
different-ai/openworkTypeScriptPublished evaluations of four automation workflows, security incidents, agent trace observability, invoice processing, and customer service, comparing Jev and frontier LLMs as structured workflows versus single prompts.
Jev Practices & PatternsDocs#official#benchmarks#evaluation
Official
Jev AI model articles
Original guides that go deeper on Jev evaluation.
Voice agents need decisions in milliseconds. How turn detection, intent, and escalation work in real time, and where a fast decision layer fits.
Jev Practices & Patterns#voice#realtime#agents
Read article
Synthetic labels can train a decision model, but they can also bake in a teacher's errors. How to generate and audit them.
Jev Practices & Patterns#synthetic-data#training#distillation
Read article
Model deprecations and price changes are operational risks. How to pin versions, read contracts, and plan a migration before you need one.
Jev Guides & Articles#operations#vendor-risk#sla
Read article
From the Jev community
Watch walkthroughs, read the threads, and see what builders are posting.
YouTube: Ray Amjad puts Jev inside an agentic coding loop and looks at what it costs to run.
YouTube: David Ondrej explains the architecture behind Jev and how to build a business on it.
YouTube: Maximilian Schwarzmüller on the new kind of model that beats LLMs in the areas it was built for.
Hacker News: Pointer to a DiffusionGemma plus vLLM tutorial for a Jev-shaped classifier.
1 pointHow to build a Jev-style classifier with DiffusionGemma and vLLMRead the thread on Hacker NewsHacker News: SafeDep Gryph plus Jev checks for compromised coding-agent behavior.
2 pointsDetecting Compromised AI Coding Agents with Jev and GryphRead the thread on Hacker NewsHacker News: Thread on Selmar's Jev code-review calibration notes and ordering.py comparisons.
Lectem2 pointsCalibrating Jev as a Code ReviewerRead the thread on Hacker NewsHacker News: Qualitative researchers classifying participant free text with Jev at negligible cost.
effisfor13 pointsI was impressed by Jev, please explain why I shouldn't beRead the thread on Hacker News