Jev directory
CommunityJev Benchmarks & Evaluations
Jev Calibration Audit — n9omi
Independent audit repository testing whether Jev's probabilities are calibrated on EIA weekly petroleum reports, comparing Jev via LangChain TypeSafeClassifier against an LLM-as-classifier, rules, and climatology baselines.
- Stars
- 0
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Checks calibration with reliability diagrams, ECE, Murphy decomposition, and the Spiegelhalter test.
- Compares Jev, an LLM-as-classifier via LangChain, rules, and climatology on identical questions.
- Ground truth comes from EIA weekly petroleum series, never from a model.
- Question types include Noul, Choice, and Score over horizons 0 and 1.
- Includes paired Brier/RPS differences, block-bootstrap CIs, and Diebold–Mariano tests.
Questions about this entry
Short answers drawn from the linked project or page.
Clone the repo, create a Python virtual environment, and run pip install -e .[jev,llm,dev]. An offline demo runs with synthetic data and no keys via jev-audit demo.
A real run needs EIA, TypeSafe, and LLM provider keys in .env. The offline demo uses synthetic data and requires no keys.
Each arm receives the same state text and typed questions from EIA-derived weekly states. For wpsr_text, you supply real EIA Weekly Petroleum Status Report highlights saved by week-ending date; dates are stripped by default.
It covers Noul yes/no, Choice, and Score questions. Horizon 0 tests current-week extraction and arithmetic; horizon 1 tests next-week draws and change buckets as forecasts.
Comparisons use paired loss differences on identical items, circular block-bootstrap confidence intervals, and Diebold–Mariano tests with Newey–West variance for autocorrelated weekly losses.
The repository is published under the GitHub account n9omi and runs as a Python package installed with pip in a virtual environment.
It is an independent audit on EIA weekly petroleum reports, not a universal evaluation of Jev. Its calibration results and recalibration maps apply to this domain and question bank, not all deployments.
Other listings of this project
Forks and copies of this project in the directory. This entry is the canonical one.
An unaffiliated, API-only audit of Jev calibration built on roughly 7,000 public HTTP calls, covering abstention, Korean, probabilistic coherence, option order, question interference, caching and latency, with a drift ledger.
Jev Practices & Patterns#community#jevbest#catalog-import
More Jev resources like this
A measurement lab for TypeSafe AI's Jev, run against api.typesafe.ai/v1/systemone with model jev-latest. It records baseline confidence scores, position-bias controls, atomic-question failures, and a 24-README semantic triage sweep, with committed data and analysis scripts.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
CommunitySuperInstance/jev-lab
A fast, typed Python framework from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models: Jev by default, Microsoft Decision-1 on OpenRouter or OpenAI's Decisions API through the same backend. It returns typed scores, thresholds and status, with optional human-label calibration.
Jev Benchmarks & Evaluations#community#github#python
Community
trustifai/typed_evalsPython
trustifai/typed_evalsPythonThe code and data behind ProbabilityBench: text-based simulated worlds with exact class probabilities, so a decision model's reported probabilities can be compared directly with ground truth. An arXiv paper and a Hugging Face dataset accompany it.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
ishida-lab/probabilitybenchPython
ishida-lab/probabilitybenchPythonIndependent experiment scoring 400 real job postings against 30 fictional résumés with Jev, producing 12,000 evaluations in about 197 seconds for roughly $0.17, with a 100-posting answer key and an analysis of what the model got wrong.
Jev Benchmarks & Evaluations#github#madewithjev#evaluation
Communitykaucap08/jev-job-matching
Controlled benchmark of Jev against Claude Sonnet 5 on one coding-agent decision: given a bug report, which file needs editing? It scores 200 real SWE-bench Verified issues with cost, latency, and confidence-band routing tables.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
CommunityK624/jev-localization-bench
Spring Boot and Spring AI benchmark comparing Java keyword rules, Jev 1.13.0, and Claude Haiku 5.5 on warehouse goods-receipt triage, with a 200-remark synthetic dataset and a Python analysis script.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
Communitymohammedahmadi1990/jev-triage
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.