maybern-tripp-smith/fedjev-bench
User repository on GitHub
FOMC hawkishness: TypeSafe Jev pairwise Choice vs rate actions (+ Haiku 4.5 comparison). Pages in /docs.
- Forks
- 1
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Benchmarks & Evaluations
Fedjev Bench
A pre-registered evaluation of whether Jev hears a rate rise coming: it shows Jev pairs of FOMC press-conference openings and asks which is more hawkish about inflation, then checks rank agreement with same-day rate moves and FedLock's scores.
- Stars
- 1
- Language
- Python
- Reading time
- 2 min
- Published by
- Community
- Author
- maybern-tripp-smith
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open sourceRequires API key
Highlights
- Run fedjev-2026-09-20 used Jev 1.13.0; Gates 1, 3, 4 and 6 PASS while Gates 2, 5 and 7 report rather than gate.
- Gate 7 is Spearman rank correlation with FedLock's published press-conference scores, not a TrueSkill re-run.
- Cached answers under runs/jev/ make the analysis offline: python scripts/analyze_gates.py needs no API key.
- README costs: main run about $0.031, name ablation +$0.011, grand total about $0.042.
- A seven-criteria multi-axis extension is documented under results/multiaxis/ with its own run id.
Quickstart
bash
git clone https://github.com/maybern-tripp-smith/fedjev-bench
cd fedjev-bench
python -m venv .venv && source .venv/bin/activate
pip install -e .
python scripts/analyze_gates.py
python scripts/plot_figures.pyWatch out
Copy reviewed against the truncated repo README on 2026-10-15; the excerpt stops inside the Khaled/jsort sensitivity section. GitHub metadata lists NOASSERTION for the licence even though the README says MIT code, and FedLock's Shah CC BY-NC 4.0 data is not fully vendored per DATA.md.
Questions about this entry
Short answers drawn from the linked project or page.
A pre-registered evaluation of TypeSafe Jev as a pairwise judge of FOMC chair press-conference openings under the criterion more hawkish about inflation. It compares Jev's orderings with same-day federal funds target changes and with FedLock's published text scores.
It answers pairwise Choice questions: given two FOMC openings, which is more hawkish about inflation. The scoreboard then asks whether those rankings recover easy hawk/dove orderings and agree with the behavioral d_same label on scheduled action days.
For run_id fedjev-2026-09-20 with jev-1.13.0, Gates 1, 3, 4 and 6 PASS; Gates 2, 5 and 7 report rather than gate. Gate 7 reports Spearman rank correlation with FedLock's published scores, raw and era-adjusted. Full tables are in REPORT.md.
The README states a main cost of about $0.031, a name ablation of +$0.011, and a grand total of about $0.042. The cached answers ship in the repo under runs/jev/, so the gate analysis itself costs nothing.
Yes for the offline analysis: clone the repo, make a venv, pip install -e ., then python scripts/analyze_gates.py and python scripts/plot_figures.py use the cached answers. A live re-score needs TYPESAFE_API_KEY and runs python score.py --live --full.
pip install -e '.[haiku]' pulls what scripts/run_haiku_comparison.py needs, and a FedLock-faithful TrueSkill replica is documented under results/fedlock_replica/. The README also documents a seven-criteria multi-axis extension with its own run id.
The README says MIT code, while GitHub metadata shows NOASSERTION. Data terms live in DATA.md and FedLock's Shah CC BY-NC 4.0 data is not fully vendored. The repo warns not to mutate the frozen gold_pairs for the recorded run id.
More Jev resources like this
A Python benchmark that evaluates LLMs' reasoning and instruction-following by making them play chess against a random player or the Komodo Dragon engine in multi-turn dialogs, separating chess skill from protocol durability.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
maxim-saplin/llm_chessPython
maxim-saplin/llm_chessPythonA worked example of using Jev as an LLM judge: ten synthetic frozen refund-support traces are scored by three yes/no criteria and one ordered helpfulness rubric in a single Jev request and recorded as a Comet Opik experiment.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
patchy631/jev-as-judgePython
patchy631/jev-as-judgePythonProphetLab's Texas Hold'em benchmark and playground for decision models: separate cash and sit-and-go leaderboards, hand replays, a Jev-backed advisor, and bring-your-own-agent tables, all with virtual chips.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
prophetlab/jevpokerbenchPython
prophetlab/jevpokerbenchPythonClassifies 1,000 recent arXiv AI papers into 24 topics with one Jev decision each, then checks those labels against an LLM judge on a seeded sample of 100. Ships three scripts and a static topic-map page.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
stas4000/jev-papersPython
stas4000/jev-papersPythonA minimal coding agent that filters every tool result through Jev before the main model reads it, plus the A/B harness that measured pass@1 and cost. The final TB4 follow-up closed the experiment without an end-to-end win.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
astro-han/jev-harnessPython
astro-han/jev-harnessPythonBenchmark comparing TypeSafe Jev 1.13 with the local Laya MLX typed-decision model across 1,470 synthetic items and eight tasks: accuracy, calibration, latency and throughput, with scripts, datasets, predictions and the full report in the repo.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
harrymunro/jev-laya-benchmarkPython
harrymunro/jev-laya-benchmarkPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.