Article
CommunityJev Integrations & Adapters
Jev vs LLM-as-a-Judge in Opik
Comet's Opik team ran Jev beside a gpt-4o-mini judge over 1,000 production turns: 89.7% agreement, 3.5x cheaper and 3.8x faster at the median, with the 103 disagreements concentrated at mid probabilities.
- Format
- Article
- Reading time
- 3 min
- Published by
- Community
- Author
- Aswin Thiyagarajan (Comet)
- Last verified
- Oct 2026
- Languages
- Python
- Platforms
- OpenRouter
- Capabilities
- Open sourceRequires API key
Highlights
- On 1,000 turns: gpt-4o-mini cost $0.0993 per 1,000 traces at 958 ms p50, and Jev cost $0.0285 at 253 ms p50, each agreeing with the other 89.7% of the time.
- Jev sends 1.5x more input tokens but is 3.5x cheaper and 3.8x faster, because the Decisions API encodes question structure alongside the text.
- 64.1% of the 103 disagreements fall between 0.3 and 0.7 on Jev's probability, against 16.7% of agreements; agreement is 100% below 0.2 and 95.6% above 0.8.
- Opik ships a native integration because the typesafe-sdk is not OpenAI-compatible: track_typesafe(TypeSafeClient()) traces every system_one call as a span.
- As an online rule Jev sits under the OpenRouter provider, takes Boolean scores only, and the span records the served version, typesafe/jev-1.13-20260917.
Watch out
Vendor-authored by Comet, whose Opik platform is the subject, and the comparison covers one narrow Boolean judging task on one team's traffic: 89.7% agreement measures consistency with the old judge, not correctness. Opik is open-source; the online judge needs a Jev model reachable under the workspace's OpenRouter key.
Questions about this entry
Short answers drawn from the linked project or page.
1,000 real turns sampled from 578 threads in Ollie's production monitoring project, re-logged so both judges scored the same turns: an existing gpt-4o-mini LLM-as-a-judge rule and Jev, both at 100% sampling with one Boolean score and the same question.
Agreement was 89.7% for both judges. Per 1,000 traces, gpt-4o-mini cost $0.0993 with 958 ms p50 and 1,202 ms p90 latency, while Jev cost $0.0285 with 253 ms p50 and 352 ms p90, despite sending 1.5x more input tokens.
The 103 split turns sit where Jev itself was unsure: 64.1% of them fall between 0.3 and 0.7 on Jev's probability versus 16.7% of agreements. Agreement is 100% below 0.2, 95.6% above 0.8, and falls to 63.5% in the 0.4-0.6 band.
Import track_typesafe from opik.integrations.typesafe and wrap TypeSafeClient with it. Every system_one call becomes an LLM span whose input is the state and questions, whose output is the answers, with token usage, provider, and cost recorded alongside.
Select it in Opik as the model behind an LLM-as-a-judge online evaluation rule under the OpenRouter provider, using your workspace key and a model such as typesafe/jev-1.13. The prompt holds only input and output, each Boolean score's description is the question, and the probability lands in the reason.
Jev has no temperature and no few-shot examples, so the wording is the whole configuration surface. Point Opik's Agent Optimizer at a labelled dataset with the candidate question as the prompt and agreement-with-label as the metric; the user template is required or it reports green with no candidates.
No, explicitly not: agreement is not correctness, and the post says which judge was right was not established. A judge also needs labels and a written policy for edge cases such as polite refusals, which the shared question scored as failures.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Microsoft-Decision-1 against Jev on a private eval
X: Vals AI's independent evaluation of Jev
X: An eval suite for System One models
X: Jev and Laya play chess instead of benchmarks
More Jev resources like this
Autonomous agent platform with 187,504 stars for building and running agents from plain English or a visual builder. Its platform features first-class TypeSafe Jev decision blocks for typed routing, filtering, scoring and confidence-gated next-action dispatching.
Jev Integrations & Adapters#community#autonomous-agents#platform
Community
Significant-Gravitas/AutoGPTPython
Significant-Gravitas/AutoGPTPythonEpistemic runtime for autonomous coding agents that uses Jev as a sub-100ms System 1 gate to shadow-evaluate capsule facts, propose evictions before bloat and prefetch memory between turns. Its fixture run reports 31.3% fewer turns and 52.6% fewer blind discovery operations, with a fail-open circuit breaker.
Jev Integrations & Adapters#community#agents#context-engineering
Community
wojciechwiesner/jit-contextPython
wojciechwiesner/jit-contextPythonA Playwright reporter that uses Jev to score which existing E2E tests a code change could affect and runs only those above a confidence threshold before browser execution, failing open to the full suite.
Jev Integrations & Adapters#community#catalog-import#logicrw
On-call tooling where Jev answers four typed questions per alert - actionable, severity, owning team and duplicate-of - and plain code routes, pages or drops. Includes a root-cause benchmark where Jev and code investigate recorded incidents with no general-purpose LLM in the loop, enforcing budgets and verification.
Jev Integrations & Adapters#community#alerting#incident-response
Community
mingleiw/jev-oncallHTML
mingleiw/jev-oncallHTMLJev-powered memory layer for AI agents that runs as a Hermes memory provider: it asks Jev to judge every eligible memory against short criteria and returns the original evidence, with Mnemosyne-compatible storage and no embedding model or vector index.
Jev Integrations & Adapters#community#catalog-import#madewithjev
Community
arslanr-com/perfectrecallPython
arslanr-com/perfectrecallPythonTwo opt-in Hermes Agent plugins that add TypeSafe Jev decision layers: a pre-tool gate for terminal, write_file, and patch calls, and a skill router that names the one skill worth loading before the model call. Stdlib only and fail-open.
Jev Integrations & Adapters#community#catalog-import#logicrw
Community
dogmatiic/hermes-jevPython
dogmatiic/hermes-jevPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.