Jev Evals

One Python interface for evaluating what LLM apps and agents say and do. Its CascadeEvaluator asks a decision model first — Jev, OpenAI's Decisions API, or any model served at /v1/systemone — keeps confident answers, and sends only the rest to an LLM judge.

A CLI from Pinecone's DevRel team for testing Agent Skills and docs against tasks: run tasks across Claude, Copilot and Gemini in local or Modal sandboxes, compare with-skill, without-skill and with-docs variants, and grade with Claude or a TypeSafe System One backend.

Jev News's desk for TypeSafe's October 14, 2026 Jev Day: Jev 1.14.0 shipped as jev-preview with opt-in images, a Box type and 262k context, plus the desk's dated API checks and what they returned.

Jev News's rollout story for Jev 1.14: OpenRouter listed typesafe/jev-1.14 while TypeSafe's jev-latest alias still returned jev-1.13.0 at 06:16 UTC, with the desk's own API calls recorded across the day.

Comet's Opik team ran Jev beside a gpt-4o-mini judge over 1,000 production turns: 89.7% agreement, 3.5x cheaper and 3.8x faster at the median, with the 103 disagreements concentrated at mid probabilities.

DeepEval's custom evaluation metric that has Jev answer bounded questions about a test case, then maps its calibrated probabilities into a deterministic weighted score, with no generated verdicts or JSON recovery.

Latitude is an open-source observability platform for AI agents: one line of telemetry captures traces, failing traces group into tracked signals, and fixes are dispatched to coding agents and replayed against real traces.

Maykana's free tool that lets a cheap AI test a web app like a person: write checks in plain sentences, Jev picks the next step from a short list, and below 80% confidence it stops and hands over to a human instead of clicking. A DeepSeek vision pass reports visual defects.

A Convex page for comparing models that answer verified multiple-choice questions about Convex, scoring them with source-weighted results and showing latency, cost and inspectable outcomes alongside the scores.

A reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.

LangChain's LangSmith team benchmarks Jev as an agent evaluator against GPT-5.6 Luna, Terra, and Claude Sonnet 4.6 on five captured weather-agent runs. Jev matched the human oracle on all 500 binary pass/fail decisions with 92-913x lower quality-score variance and cost $0.00035 per call.