Jev directory
CommunityJev Benchmarks & Evaluations
LLM As Jev Dataset
Public Hugging Face dataset by Hasnain Akhtar holding Jev-style decision items with source, state, options, and label fields, alongside declared calibration metrics such as accuracy, ECE, and Brier score. The dataset viewer is unavailable because generation failed on a schema cast.
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Hosted on Hugging Face as Hasnain1241/llm-as-jev-dataset; public and not gated.
- Declared features include acc, ece, brier, n, and text_mode parse statistics.
- Row data appears as lists of structs with source, state, options, and label fields.
- Dataset viewer is unavailable: generation failed with a TypeError on a schema cast.
- Splits listed under the default subset: train and test.
Questions about this entry
Short answers drawn from the linked project or page.
It is a public dataset repository published by user Hasnain1241 (Hasnain Akhtar) that holds Jev-style decision records with source, state, options, and label fields, plus a declared set of calibration metrics.
The viewer reports DatasetGenerationError. The underlying exception is a TypeError: the row data is a list of structs with source, state, options, and label, which could not be cast to the declared feature schema of acc, ece, brier, n, and text_mode.
The row data shown in the error trace is a list of structs with four fields: source (string), state (string), options (list of strings), and label (int64).
The declared schema lists acc and ece and brier as float64 values, n as int64, and a text_mode struct with n, parse_failures, argmax_match, and match_rate_of_parsed.
No. The repository metadata reports isGated false and isPrivate false, so the dataset is publicly listed on Hugging Face.
The metadata reports hasParquetFormat false, and the parquet and info job failed with the same cast error, so no converted parquet branch was produced.
It is maintained by the Hugging Face user Hasnain1241, listed as Hasnain Akhtar, a non-Pro individual account with one follower. No organization is attached to the repository.
More Jev resources like this
Controlled benchmark of Jev against Claude Sonnet 5 on one coding-agent decision: given a bug report, which file needs editing? It scores 200 real SWE-bench Verified issues with cost, latency, and confidence-band routing tables.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
CommunityK624/jev-localization-bench
Independent audit repository testing whether Jev's probabilities are calibrated on EIA weekly petroleum reports, comparing Jev via LangChain TypeSafeClassifier against an LLM-as-classifier, rules, and climatology baselines.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
Communityn9omi/jev-calibration-audit
A measurement lab for TypeSafe AI's Jev, run against api.typesafe.ai/v1/systemone with model jev-latest. It records baseline confidence scores, position-bias controls, atomic-question failures, and a 24-README semantic triage sweep, with committed data and analysis scripts.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
CommunitySuperInstance/jev-lab
A fast, typed Python framework from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models: Jev by default, Microsoft Decision-1 on OpenRouter or OpenAI's Decisions API through the same backend. It returns typed scores, thresholds and status, with optional human-label calibration.
Jev Benchmarks & Evaluations#community#github#python
Community
trustifai/typed_evalsPython
trustifai/typed_evalsPythonThe code and data behind ProbabilityBench: text-based simulated worlds with exact class probabilities, so a decision model's reported probabilities can be compared directly with ground truth. An arXiv paper and a Hugging Face dataset accompany it.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
ishida-lab/probabilitybenchPython
ishida-lab/probabilitybenchPythonBenchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.