Organization repository on GitHub
ProbabilityBench: Evaluating Jev and Other Decision Models on True Probabilities
- Forks
- 1
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.Homepage
Jev directory
CommunityJev Benchmarks & Evaluations
ProbabilityBench (code)
The code and data behind ProbabilityBench: text-based simulated worlds with exact class probabilities, so a decision model's reported probabilities can be compared directly with ground truth. An arXiv paper and a Hugging Face dataset accompany it.
- Stars
- 10
- Language
- Python
- License
- MIT
- Reading time
- 2 min
- Published by
- Community
- Author
- ishida-lab
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open source
Highlights
- Provides text-based simulated worlds with exact class probabilities, unlike one-outcome real datasets.
- Each generated case has a readable memo, a sampled outcome and the exact positive probability.
- Paper: arXiv 2610.17016; dataset published on Hugging Face as ishidalab/probabilitybench.
- Links a blog post at ishida-lab.github.io and is MIT licensed per the README badge.
- Topics: bayes-error, benchmark, calibration, decision-model, jev.
Watch out
README and GitHub metadata reviewed 2026-10-17. The excerpt truncates inside the worlds table and does not list task names, model results or Jev-specific scores; the companion arXiv paper is already catalogued separately in this directory.
Questions about this entry
Short answers drawn from the linked project or page.
Real-world datasets usually provide only one observed outcome per input, so the true conditional probability is unknown and model uncertainty is hard to evaluate. ProbabilityBench generates simulated worlds where exact class probabilities are known.
A repo topic is jev and the benchmark evaluates decision models on true probabilities; the digest excerpt does not list Jev's per-task scores or the exact model version tested, so those are not reported here.
Each case includes a readable memo, a sampled outcome, and the exact probability of the positive outcome given what the memo shows, so reported probabilities can be compared directly with ground truth.
The README links arXiv 2610.17016 for the paper, a Hugging Face dataset at ishidalab/probabilitybench, and a blog post at ishida-lab.github.io/blog_probabilitybench.html.
No: it consists of simulated worlds with ex ante known probabilities. That makes ground truth exact, but it means the benchmark tests probability reporting quality rather than uncertainty on naturally occurring text.
The digest shows a table header listing worlds and task names but truncates before the rows, so the full set of simulated worlds is not evidenced here; check the repo or the paper for the complete list.
The ishida-lab organisation publishes the repository under MIT; digest metadata shows 10 stars, 1 fork, creation 2026-10-14 and a push on 2026-10-15.
More Jev resources like this
A timestamped benchmark and trading arena that replays resolved prediction markets: models commit to probabilities using only information available at the time, and realized P&L is the headline metric with Brier, log-loss and ECE as checks.
Jev Benchmarks & Evaluations#community#github#python
Community
ruyianry/JevGymPython
ruyianry/JevGymPythonA benchmarking studio and real-time simulator that pits Jev (remote TypeSafe System One), a local Laya model and the optimal blackjack Basic Strategy against each other on identical deterministic shoes. It scores every deviation turn by turn and runs headless batches of up to 100,000 rounds.
Jev Benchmarks & Evaluations#community#github#blackjack
Community
mrbanana55/decision_blackjackPython
mrbanana55/decision_blackjackPythonGTM workflow experiments with Jev from deepdave98: weekly datasets, prompts, and benchmark results that test Jev against Claude on lead triage and routing, deal risk, reply triage, MEDDPICC from call summaries, duplicate accounts, and closed-lost reactivation.
Jev Benchmarks & Evaluations#community#github#gtm
Community
deepdave98/jev-playgroundPython
deepdave98/jev-playgroundPythonA fast, typed Python framework from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models: Jev by default, Microsoft Decision-1 on OpenRouter or OpenAI's Decisions API through the same backend. It returns typed scores, thresholds and status, with optional human-label calibration.
Jev Benchmarks & Evaluations#community#github#python
Community
trustifai/typed_evalsPython
trustifai/typed_evalsPythonA benchmark of Jev and other decision models on trolley-problem states, with runners, raw answers and analysis in one repository. The v1 snapshot covers 1,098 states and the October 2026 v2 run has 14 models on 5,240 frozen states.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
zachlandes/trolley-benchHTML
zachlandes/trolley-benchHTMLAkitaOnRails' study documentation and benchmark harness for Jev-like decision models: frozen inputs, inspectable expected answers, raw response archives and reproducible scoring compare TypeSafe Jev with Decision-1, OpenAI Decisions, Clef, Decision 3.0, Laya, Kev, Eikos, Jebadiah, Mapika Decider and GLiNER.
Jev Benchmarks & Evaluations#community#github#python
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.