User repository on GitHub
Trolley bench v1: Jev and nine open decision models on 1,098 trolley-problem states, with runners, raw answers and analysis
- Forks
- 0
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Benchmarks & Evaluations
Trolley benchmark
A benchmark of Jev and other decision models on trolley-problem states, with runners, raw answers and analysis in one repository. The v1 snapshot covers 1,098 states and the October 2026 v2 run has 14 models on 5,240 frozen states.
- Stars
- 14
- Language
- HTML
- Reading time
- 2 min
- Published by
- Community
- Author
- zachlandes
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open source
Highlights
- v1 snapshot: Jev and nine open decision models on 1,098 trolley-problem states.
- October 2026 v2 run: 14 models answer the same three yes/no questions about 5,240 frozen states.
- Hosted models are Jev (jev-1.13.0) and GPT-6 Luna; the other 12 run locally with pinned weights.
- Includes prompts, runtime recipes, every full run's raw answers and static result explorers.
- Rebuild needs only Python 3.10+, Node.js 22+ standard libraries and git; no API key required.
Quickstart
bash
python3 scripts/rebuild.py --checkWatch out
README and GitHub metadata reviewed 2026-10-17. The excerpt truncates the evidence-inputs paragraph, so per-model scores are not captured here; the repo license field is blank in the digest metadata.
Questions about this entry
Short answers drawn from the linked project or page.
A repository containing the v1 snapshot (Jev and nine open decision models on 1,098 trolley-problem states) and an October 2026 v2 benchmark in which 14 models answer the same three yes/no questions about 5,240 frozen trolley states.
Jev is one of the two hosted models, identified as jev-1.13.0 (alongside GPT-6 Luna); the other 12 models run locally with pinned weights and runtime recipes. The digest does not include Jev's final scores.
The prompts, pinned local weights and runtime recipes, every full run's raw answers, the analysis, and factual static result explorers, plus verified download receipts and curated model labels.
Clone the repository and run python3 scripts/rebuild.py --check with Python 3.10+ and Node.js 22+ (standard libraries only) plus git. The --check flag compares regenerated tracked files against git HEAD and exits nonzero if any differ.
No. The README states no API key is needed to rebuild the published results; the raw request rows, summaries and provenance receipts are checked into the repository as evidence inputs.
Only two hosted models are named in the digest: Jev at jev-1.13.0 and GPT-6 Luna at gpt-6-luna; the other twelve run locally from weights pinned in the repository.
GitHub user zachlandes created the repository 2026-09-29 and pushed it 2026-10-15, per the digest metadata; the license field is blank and no funding or affiliation is stated there.
More Jev resources like this
AkitaOnRails' study documentation and benchmark harness for Jev-like decision models: frozen inputs, inspectable expected answers, raw response archives and reproducible scoring compare TypeSafe Jev with Decision-1, OpenAI Decisions, Clef, Decision 3.0, Laya, Kev, Eikos, Jebadiah, Mapika Decider and GLiNER.
Jev Benchmarks & Evaluations#community#github#python
A Python experiment that adapts Gemma 4 E4B for System One-style decisions: one call takes a situation and a fixed answer list and returns a probability per answer, with per-dataset adapters and held-out results across four datasets.
Jev Benchmarks & Evaluations#community#github#python
Community
omnaiduu/jevPython
omnaiduu/jevPythonA study of script-dependent answers about Taiwan in commercial decision models: 28 statements asked in Traditional Chinese, Simplified Chinese and English to OpenAI, TypeSafe Jev and Cloudflare models, with data, code and a paper.
Jev Benchmarks & Evaluations#community#github#research
The code and data behind ProbabilityBench: text-based simulated worlds with exact class probabilities, so a decision model's reported probabilities can be compared directly with ground truth. An arXiv paper and a Hugging Face dataset accompany it.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
ishida-lab/probabilitybenchPython
ishida-lab/probabilitybenchPythonA reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.
Jev Benchmarks & Evaluations#community#python#evals
Community
keta1930/what-the-jevPython
keta1930/what-the-jevPythonA benchmark harness and web UI that cross-compares dozens of zero-shot information-extraction and classification systems — extractor encoders, rerankers, and typed-decision engines like Jev — on accuracy, latency, and cost.
Jev Benchmarks & Evaluations#community#python#benchmark
Community
umstek/zero-shot-ie-benchPython
umstek/zero-shot-ie-benchPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.