How does it handle training-data leakage of rulings?

It narrows one path by using cutoff-eligible decisions, removing outcome-leaking material, and running without web access. The author states this reduces leakage but does not prove zero contamination.

Why forecast rulings instead of testing bar-exam skills?

Junior-task tests miss senior judgment, so this asks models to predict grant or denial per claim and defendant from the judge's record. The ground truth is objective even though the judgment is hard.

What exactly is scored and how should I read it?

The unit is the claim-defendant pair with micro-Brier scoring and intervals clustered by case family. The first cycle compares models on a shared frozen record rather than fitting baselines.

Can I reproduce the pipeline without official cases?

Run the synthetic fixture to prove the pipeline works, then read the methods, reproduce guide, and publication rules. Fixture outputs are plumbing checks, not public benchmark results.

How are LegalForecastBench official runs executed?

Each official run is a GitHub Actions matrix job with one cell per model and case, isolating failures and producing a per-cell audit trail; repeat_count must be exactly 1 until repeated-sampling fan-in is supported.

What is the prediction unit in LegalForecastBench?

The claim-defendant pair rather than the motion as a whole, predicting the probability that each challenged claim against each challenged defendant is dismissed in full, scored as micro-Brier with clustered intervals.

How do I run the LegalForecastBench pipeline without the official cases?

uv run legalforecast run issue-fixture --output-dir tmp/fixture-run runs the synthetic fixture end to end and writes forecast, label, packet, prompt and model-registry files; the README states those prove the pipeline works and are not public benchmark results.

CommunityJev Practices & Patterns

LegalForecastBench

Benchmark harness that tests whether models can forecast federal motion-to-dismiss outcomes from the written record, scored per claim-defendant pair with micro-Brier and clustered intervals.

See highlights

Open github.com

Stars: 7
Language: Python
License: APACHE-2.0
Published by: Community
Author: johnhughes3
Last verified: Oct 2026

johnhughes3/LegalForecastBench

Organization repository on GitHub

View on GitHub

LegalForecast-MTD benchmark alpha and official evaluation workflows

Forks: 1
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.Homepage

About this project

LegalForecast-MTD tests whether frontier models can forecast federal motion-to-dismiss rulings from the judge's written record. It reports claim-defendant micro-Brier scores with clustered intervals. Eligible decisions follow the models' latest first external deployment; known outcome leakage is excluded, and official models run without web access. This reduces a defined leakage path; it does not prove zero contamination or frozen served weights.

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import awesomejev

Works forJev for Classification Jev for Evaluation

Used inJev for Software & DevTools

Languages: Python

Highlights

Predicts dismissal probability per challenged claim and defendant from the same written record.
Controls contamination with cutoff eligibility, redacted inputs, and no-network official runs.
Executes official runs as isolated per-model-per-case jobs with bound state and authenticated reruns.
Separates official cycles from community fixture comparisons with explicit contributor and adapter docs.

Watch out

Official model rows are withheld while the full report is assembled, and cohort scores are not comparable across releases. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-on-a-laptop

Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.