Jev directory
CommunityJev Benchmarks & Evaluations
EvalSafe Customer Service
Hugging Face dataset from TypeSafe AI of customer-service dialogs with per-policy decisions from OpenAI and Anthropic models and a mean-probability consensus field. Parquet, English, 1K–10K rows.
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Dialogs carry customer, account, and state fields such as intent, priority, and handed_off.
- Each case includes decisions for policies A, B, and C from two providers.
- Consensus is computed with the mean_probabilities method across contributors.
- Actions include respond_with, handoff_human, set_intent, and set_priority.
- Rows are parquet, English, 1K–10K in size, loadable with Datasets, pandas, or Polars.
Questions about this entry
Short answers drawn from the linked project or page.
It is hosted on Hugging Face as parquet files with the Datasets, pandas, and Polars libraries listed, so it can be loaded with the datasets library or read directly as parquet.
A row carries a case_id, dialog_id, turn number, the transcript so far, customer and account fields, a state object, a metadata note, provider decision JSON, and a consensus block.
Each case includes decisions for policy_id A, B, and C, produced separately by the OpenAI and Anthropic model outputs recorded in the row.
The consensus block reports method mean_probabilities with contributors openai and anthropic, and a used_opus_fallback flag set to false in the sample rows.
Actions seen in the rows include respond_with with a message key, handoff_human with a queue of general or supervisor, set_intent with account_access, and set_priority with urgent.
Metadata includes dialog_id, turn, and a theme such as access, plus a note describing the expected close call, for example urgency between 1 and 2 where one policy's threshold is the deciding factor.
The rows record openai model gpt-6-astra and anthropic model claude-fable-5-1, both at high reasoning effort with question_mode one and used_opus_fallback false.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Facebook: Intake Router keeps diagnosis out of the answer space at the front desk.
FacebookClinicnotes
X: OpenRouter Decision Model Rankings
X: herdr + jev, an agent router
X: A group chat bot with decision room routing
More Jev resources like this
Controlled benchmark of Jev against Claude Sonnet 5 on one coding-agent decision: given a bug report, which file needs editing? It scores 200 real SWE-bench Verified issues with cost, latency, and confidence-band routing tables.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
CommunityK624/jev-localization-bench
Hugging Face dataset of run results for the kieve-jev-fractal-300m decision engine, recording per-benchmark tallies of requests, answered, abstained and errored cases along with scores, latency medians and p95s, and retrieval and cost-aware metrics.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
Community
Benchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
Spring Boot and Spring AI benchmark comparing Java keyword rules, Jev 1.13.0, and Claude Haiku 5.5 on warehouse goods-receipt triage, with a 200-remark synthetic dataset and a Python analysis script.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
Communitymohammedahmadi1990/jev-triage
A TypeSafe AI Hugging Face dataset of agent trace observability records for evaluation workflows, with cases (111 rows), questions (1.12k rows), and run_results (999 rows) in parquet, including per-provider and consensus decision blocks.
Jev Benchmarks & Evaluations#evaluation#agents#benchmark
Community
Hugging Face dataset from TypeSafe AI pairing security incident telemetry with model decision outputs. Tabular and text parquet rows, English, 1K–10K records, loadable with Datasets, pandas, or Polars.
Jev Benchmarks & Evaluations#evaluation#benchmark#security
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.