Jev directory
CommunityJev Benchmarks & Evaluations
EvalSafe Agent Trace Observability Dataset
A TypeSafe AI Hugging Face dataset of agent trace observability records for evaluation workflows, with cases (111 rows), questions (1.12k rows), and run_results (999 rows) in parquet, including per-provider and consensus decision blocks.
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Three subsets: cases (111 rows), questions (1.12k rows), and run_results (999 rows).
- Stored as parquet with tabular and text modalities; language listed as English.
- Traces carry workflow_name, group_id, start and end timestamps, and metadata such as env, deployment_id, and agent_version.
- Each trace includes openai, anthropic, and consensus decision blocks with policy_id, status, and actions.
- Consensus rows use method mean_probabilities with contributors openai and anthropic.
Questions about this entry
Short answers drawn from the linked project or page.
It holds agent trace observability records for evaluation workflows, split into three subsets: cases with 111 rows, questions with 1.12k rows, and run_results with 999 rows. The single listed split is test with 111 rows.
The dataset is stored as parquet and auto-converted to Parquet for the viewer. It is tagged with the Datasets, pandas, and Polars libraries, so it can be loaded through the Hugging Face Datasets tooling or read directly as parquet.
The listing reports a size band of 1K to 10K rows. The subsets break down as cases at 111 rows, questions at 1.12k rows, and run_results at 999 rows, with a test split of 111 rows.
Traces carry an object type and id, a workflow_name, a group_id, started_at and ended_at timestamps, and metadata such as env, deployment_id, agent_version, and policy_version.
Decision blocks reference gpt-6-astra from openai and claude-fable-5-1 from anthropic, both with reasoning_effort set to high and question_mode set to one. Consensus blocks combine the two providers.
Consensus rows use the method mean_probabilities with contributors listed as openai and anthropic. Each consensus entry also carries decisions with a policy_id, a status, and named actions such as human_review or page_on_call.
It is published by the TypeSafe AI organization on Hugging Face, which the listing shows with 113 followers and 4 likes on this dataset. The digest does not state a license, so check the dataset card for terms.
More Jev resources like this
Benchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
Scala 3 benchmark running one Jackson documentation-research task over an MCP catalog through three designs: a direct LLM agent, an LLM with a nested Jev evidence tool, and top-level Jev orchestration, comparing turns, MCP calls, token use and latency.
Jev Benchmarks & Evaluations#benchmark#evaluation#mcp
Communityjamesward/hello-zio-typesafe-ai
Controlled benchmark of Jev against Claude Sonnet 5 on one coding-agent decision: given a bug report, which file needs editing? It scores 200 real SWE-bench Verified issues with cost, latency, and confidence-band routing tables.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
CommunityK624/jev-localization-bench
Spring Boot and Spring AI benchmark comparing Java keyword rules, Jev 1.13.0, and Claude Haiku 5.5 on warehouse goods-receipt triage, with a 200-remark synthetic dataset and a Python analysis script.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
Communitymohammedahmadi1990/jev-triage
arXiv paper evaluating 22 hosted decision models and two prompted general models as runtime classifiers for a multi-rule coding-agent policy engine. It reports recall, false-positive, latency, and cost results on a 2,643-call synthetic dataset with a sealed holdout.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Hugging Face dataset from TypeSafe AI of customer-service dialogs with per-policy decisions from OpenAI and Anthropic models and a mean-probability consensus field. Parquet, English, 1K–10K rows.
Jev Benchmarks & Evaluations#evaluation#benchmark#routing
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.