User repository on GitHub
JevGym is an open-source platform designed to benchmark and facilitate better probabilistic estimation in Jev-alike models
- Forks
- 1
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.Homepage
Jev directory
CommunityJev Benchmarks & Evaluations
JevGym
A timestamped benchmark and trading arena that replays resolved prediction markets: models commit to probabilities using only information available at the time, and realized P&L is the headline metric with Brier, log-loss and ECE as checks.
- Stars
- 104
- Language
- Python
- Reading time
- 2 min
- Published by
- Community
- Author
- ruyianry
- Last verified
- Oct 2026
Used inJev for Finance & Fintech
- Languages
- Python
- Capabilities
- Open source
Highlights
- Replays real, already-resolved prediction markets so a model forecasts the past as if it were the present.
- Leakage is prevented by construction: every evidence item carries a public-availability timestamp and available_at <= snapshot.timestamp is enforced.
- Realized P&L from the trading arena is the headline metric; Brier, log-loss and ECE are the calibration checks.
- Open to any model, not just Jev: official Jev, open Jev-wire servers, a fine-tuned Qwen or a plain LLM, over identical markets and prompts.
- On clearer Kalshi markets the README reports official Jev averaging +11.8% return per contract across weather, crypto and the Fed.
Watch out
README reviewed 2026-10-17; the excerpt is truncated mid-sentence in the Kalshi numbers section, GitHub metadata lists the licence as NOASSERTION, and the +11.8% figure is the project's own result rather than an independent evaluation.
Questions about this entry
Short answers drawn from the linked project or page.
A timestamped benchmark and trading arena for real-world probabilistic forecasting that replays already-resolved prediction markets: a model sees the information available at a past moment, commits to a probability, and is scored against what happened.
It measures the Jev-series (System One) decision models but is open to any model: official Jev, open Jev-wire servers, a fine-tuned Qwen or a plain LLM, all over the same markets, the same frozen prompt and the same scoring.
Every piece of evidence carries a public-availability timestamp and the rule available_at <= snapshot.timestamp is enforced; agents that search the web are bounded to the same moment via the Internet Archive.
Realized P&L: each forecast becomes a trade against the contemporaneous market price. Calibration metrics, Brier score, log-loss and ECE, are the check, and the market crowd is a baseline, never the oracle.
On real resolved Kalshi markets the README reports Jev profitable with +11.8% average return per contract on the clearer markets across weather, crypto and the Fed, carrying signal before a contract is even priced.
The README links a project site at ruyianry.github.io/JevGym, an interactive demo called can you beat Jev?, and a KALSHIJEV.md component file. The digest excerpt does not show setup commands or data requirements.
GitHub reports the licence as NOASSERTION, meaning no standard licence was detected. The Python repo has 104 stars and 1 fork, created 2026-09-22 and last pushed 2026-09-23.
More Jev resources like this
The code and data behind ProbabilityBench: text-based simulated worlds with exact class probabilities, so a decision model's reported probabilities can be compared directly with ground truth. An arXiv paper and a Hugging Face dataset accompany it.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
ishida-lab/probabilitybenchPython
ishida-lab/probabilitybenchPythonGTM workflow experiments with Jev from deepdave98: weekly datasets, prompts, and benchmark results that test Jev against Claude on lead triage and routing, deal risk, reply triage, MEDDPICC from call summaries, duplicate accounts, and closed-lost reactivation.
Jev Benchmarks & Evaluations#community#github#gtm
Community
deepdave98/jev-playgroundPython
deepdave98/jev-playgroundPythonA benchmarking studio and real-time simulator that pits Jev (remote TypeSafe System One), a local Laya model and the optimal blackjack Basic Strategy against each other on identical deterministic shoes. It scores every deviation turn by turn and runs headless batches of up to 100,000 rounds.
Jev Benchmarks & Evaluations#community#github#blackjack
Community
mrbanana55/decision_blackjackPython
mrbanana55/decision_blackjackPythonA benchmark of Jev and other decision models on trolley-problem states, with runners, raw answers and analysis in one repository. The v1 snapshot covers 1,098 states and the October 2026 v2 run has 14 models on 5,240 frozen states.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
zachlandes/trolley-benchHTML
zachlandes/trolley-benchHTMLAkitaOnRails' study documentation and benchmark harness for Jev-like decision models: frozen inputs, inspectable expected answers, raw response archives and reproducible scoring compare TypeSafe Jev with Decision-1, OpenAI Decisions, Clef, Decision 3.0, Laya, Kev, Eikos, Jebadiah, Mapika Decider and GLiNER.
Jev Benchmarks & Evaluations#community#github#python
A four-step experiment asking whether Jev, a fast model that picks among named options, can act as the policy in StarCraft multi-agent micro (SMAC), comparing every step against baselines that had the same information but no Jev.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
wang90063/jev-on-smacPython
wang90063/jev-on-smacPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.