Jev directory
CommunityJev Benchmarks & Evaluations
EvalSafe OneT
Hugging Face dataset from TypeSafe AI for evaluating decision models on workplace documents. Three subsets cover 150 cases, 7.5k questions and 67.5k run results, distributed as parquet in English under an Apache-2.0 license.
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Three subsets: cases (150 rows), questions (7.5k rows) and run_results (67.5k rows).
- Single test split of 150 rows; dataset size listed as 10K-100K.
- Parquet files loadable with Datasets, pandas and Polars.
- Apache-2.0 license; English text and tabular modalities.
- Case metadata records occupation_title, persona_title and document shape.
Questions about this entry
Short answers drawn from the linked project or page.
Three subsets: cases with 150 rows, questions with 7.5k rows, and run_results with 67.5k rows. The dataset has a single test split holding 150 rows.
The dataset card lists an Apache-2.0 license. It is published by the TypeSafe AI organization on Hugging Face and tagged as an evaluation synthetic benchmark.
The card lists Datasets, pandas and Polars. Files are in parquet format and are auto-converted to Parquet for the Dataset Viewer and Data Studio.
Each case row carries case_id, input_json, metadata_json and question_instance_ids. The metadata includes occupation_title, persona_title and a shape description of the document type.
Examples include performance evaluation reports for regional sales managers, employee evaluation and disciplinary action forms, requests for proposal, training budget variance reports, childcare daily reports and staffing requirement memos.
The card lists English as the only language and a size range of 10K to 100K rows, with tabular and text modalities.
Yes. The Hugging Face Dataset Viewer and Data Studio render the parquet files in the browser, and a SQL console is available for querying the subsets directly.
More Jev resources like this
A Python benchmark that evaluates LLMs' reasoning and instruction-following by making them play chess against a random player or the Komodo Dragon engine in multi-turn dialogs, separating chess skill from protocol durability.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
maxim-saplin/llm_chessPython
maxim-saplin/llm_chessPythonIndependent experiment scoring 400 real job postings against 30 fictional résumés with Jev, producing 12,000 evaluations in about 197 seconds for roughly $0.17, with a 100-posting answer key and an analysis of what the model got wrong.
Jev Benchmarks & Evaluations#github#madewithjev#evaluation
Communitykaucap08/jev-job-matching
Controlled benchmark of Jev against Claude Sonnet 5 on one coding-agent decision: given a bug report, which file needs editing? It scores 200 real SWE-bench Verified issues with cost, latency, and confidence-band routing tables.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
CommunityK624/jev-localization-bench
Independent audit repository testing whether Jev's probabilities are calibrated on EIA weekly petroleum reports, comparing Jev via LangChain TypeSafeClassifier against an LLM-as-classifier, rules, and climatology baselines.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
Communityn9omi/jev-calibration-audit
Public Hugging Face dataset by Hasnain Akhtar holding Jev-style decision items with source, state, options, and label fields, alongside declared calibration metrics such as accuracy, ECE, and Brier score. The dataset viewer is unavailable because generation failed on a schema cast.
Jev Benchmarks & Evaluations#calibration#evaluation#benchmark
Community
A measurement lab for TypeSafe AI's Jev, run against api.typesafe.ai/v1/systemone with model jev-latest. It records baseline confidence scores, position-bias controls, atomic-question failures, and a 24-README semantic triage sweep, with committed data and analysis scripts.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
CommunitySuperInstance/jev-lab
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.