Jev directory
CommunityJev Benchmarks & Evaluations
Jev-RAG-Claim-Verification-130K
Hugging Face dataset of 130k rows pairing RAG claim-verification and response-quality questions with Jev-1.13.0 predictions, gold labels, confidences, token counts, latency, and cost. Derived from wandb/RAGTruth-processed.
- Reading time
- 2 min
- Published by
- Community
- Added
- Oct 2026
Highlights
- 130k rows split into 126k train, 2.09k validation, and 2.06k test
- Text-classification dataset stored as parquet, English, tabular text modality
- Columns include jev_prediction, jev_confidence, jev_probabilities_json, and jev_latency_ms
- Task families include rag_claim_verification and rag_response_quality
- Derived from wandb/RAGTruth-processed using transforms at version 0.1.0
Questions about this entry
Short answers drawn from the linked project or page.
It is a Hugging Face dataset with parquet files and three splits: train (126k rows), validation (2.09k rows), and test (2.06k rows). Load it with the Datasets library or read the parquet files directly; the page also offers a Dataset Viewer and SQL console.
The dataset card lists the license as other. The source_license column records mit-with-upstream-review-required for rows derived from wandb/RAGTruth-processed, so check the upstream terms before redistribution.
Rows carry question_text, criteria_json, state_json, request_json, gold_label, gold_score, and label_source, alongside Jev output fields such as jev_prediction, jev_confidence, jev_probabilities_json, jev_answer_json, and jev_error.
The jev_model column records jev-1.13.0 for the sampled rows, and jev_request_id, jev_completed_at, and jev_raw_response_json capture the individual request metadata.
Sampled rows use a noul type for the fully_grounded question and a choice type for the quality question. The fully_grounded question asks whether every factual claim in the assistant response is supported by the authoritative context.
Yes. Columns include jev_input_tokens, jev_output_tokens, jev_estimated_cost_usd, and jev_latency_ms. One sampled row reports 1,115 input tokens, 24 output tokens, 645 ms latency, and an estimated cost of 0.000047 USD.
It is published by the Hugging Face user suryatmodulus and is marked as duplicated from MaziyarPanahi/Jev-RAG-Claim-Verification-130K. Rows derive from wandb/RAGTruth-processed via transformations such as ragtruth_response_to_noul and ragtruth_quality_to_choice.
More Jev resources like this
Benchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
Spring Boot and Spring AI benchmark comparing Java keyword rules, Jev 1.13.0, and Claude Haiku 5.5 on warehouse goods-receipt triage, with a 200-remark synthetic dataset and a Python analysis script.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
Communitymohammedahmadi1990/jev-triage
Independent experiment scoring 400 real job postings against 30 fictional résumés with Jev, producing 12,000 evaluations in about 197 seconds for roughly $0.17, with a 100-posting answer key and an analysis of what the model got wrong.
Jev Benchmarks & Evaluations#github#madewithjev#evaluation
Communitykaucap08/jev-job-matching
Controlled benchmark of Jev against Claude Sonnet 5 on one coding-agent decision: given a bug report, which file needs editing? It scores 200 real SWE-bench Verified issues with cost, latency, and confidence-band routing tables.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
CommunityK624/jev-localization-bench
A measurement lab for TypeSafe AI's Jev, run against api.typesafe.ai/v1/systemone with model jev-latest. It records baseline confidence scores, position-bias controls, atomic-question failures, and a 24-README semantic triage sweep, with committed data and analysis scripts.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
CommunitySuperInstance/jev-lab
ML Intern Lab's benchmark write-up of open decision models on the LocalLLaMA/typed-decisions test split: 2,000 decisions per model for GEV-26B-Decide, JEV-35B and pplx-decider-v1.1-27b, with accuracy, KL, Brier, ECE and latency tables. The repository states no licence.
Jev Benchmarks & EvaluationsDocs#community#evaluation#benchmark
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.