Article
CommunityJev Benchmarks & Evaluations
Comparing Jev, Gemini, and OpenAI Decision Models
SlopCop blog post by Jonathan Ellis, published November 12, 2026, reporting a bakeoff of Jev against Gemini Decisions and OpenAI for classifying GitHub event relevance, the switch to GLiDE from Fastino Labs, and lessons on eval design.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- SlopCop ran a bakeoff of Jev, Gemini Decisions, and OpenAI for classifying GitHub event relevance.
- The team chose GLiDE from Fastino Labs for its revision, citing best results for its use case and low cost.
- GLiDE was the second-slowest service tested; Gemini and OpenAI most excelled on inference speed.
- Jev's recall looked like 0.91 against incumbent-derived labels but 0.57 against an independent panel.
- Relevant pairs were 11% of traffic, so the eval stratified on the incumbent's verdict and an independent Voyage reranker score.
Questions about this entry
Short answers drawn from the linked project or page.
It is a bakeoff of Jev, Google's Gemini Decisions, and OpenAI decision models for classifying whether GitHub events are relevant to a developer's work, written by Jonathan Ellis and published November 12, 2026.
They chose GLiDE from Fastino Labs for their revision, saying it gave the best results for their use case and was one of the cheapest options, with open-ish weights that require a separate commercial agreement above $10M annual revenue.
Against labels built from the incumbent's verdicts, Jev's recall looked like 0.91; against an independent panel it was 0.57. The post attributes the gap to truth filtered through the incumbent's answers flattering the incumbent.
GLiDE was the second-slowest service tested, while Gemini and OpenAI most excelled on inference speed. The post credits those vendors' massive hardware footprints and inference-speed expertise.
Relevant pairs were 11% of traffic, so a uniform sample would waste most labels on easy negatives. They stratified on the incumbent's verdict and on an independent Voyage reranker score, then weighted each pair by its stratum's size.
It says Jev is a good place to start when you lack the time or data for a proper eval, describing it as inexpensive, smart, and reasonably fast, while noting you can probably do better depending on your domain and priorities.
Recording why a pair is relevant, not only whether, showed that every model loses most of its recall on dependency items. That shifted the follow-up work from choosing a model to improving evidence.
More Jev resources like this
Benchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
CodeCut article by Khuyen Tran comparing TypeSafe Jev, RapidFuzz, and Sentence Transformers on the Abt-Buy product matching benchmark, with per-tool merge counts and a walkthrough of asking Jev which product is the same.
Jev Benchmarks & EvaluationsArticle#evaluation#benchmark#classification
Community
arXiv paper presenting AbductionBench, a unified framework and taxonomy for abductive reasoning tasks, instantiated on 27 datasets across ten domains and used to evaluate eight LLMs from four model families on single-choice explanation selection.
Jev Benchmarks & EvaluationsArticle#arxiv#benchmark#evaluation
Community
Open, deterministic audit benchmark for System 1 decision models, covering calibration under label-preserving perturbations, cross-question interference, and post-hoc attribution faithfulness across five open-weight models and generative baselines.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper evaluating 67 systems, including 47 decision models and 20 LLMs, on 43 Arabic tasks across 9 task families under a shared zero-shot protocol. Jev 1.14.0 scores 69.24 against 68.77 for Jev 1.13.0 and 71.81 for Gemma-4-31B.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Multilingual benchmark of nine suites in 122 languages that evaluates typed decision models such as Jev, Clef, and pplx-decider, and tests whether confidence thresholds fitted on English transfer to other languages.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.