Do I need a Jev API key, or can I run it fully local?

Jev via TypeSafe is one judge backend; local models through Ollama, vLLM, or LM Studio, an ensemble committee, and a deterministic mock judge are also supported. The mock path runs offline for zero-cost testing.

How does it avoid comparing every pair in a large set?

The Swiss matchmaker pairs items with adjacent ratings that have not met yet, targeting maximum-information duels instead of all-pairs round-robin. That is how a 10,000-item set drops to roughly 70,000 matches.

What stops position bias and slow rating convergence?

Presentation order is coin-flipped per duel and mapped back to contestant IDs, countering first-candidate favouritism. Elo uses a decaying K-factor starting high for placement then tightening.

What do I actually run to start a tournament?

Run the FastAPI server for the WebSocket dashboard with selectable engine and judge, or drive main.py from the CLI with dataset, rounds, and concurrency flags. Results persist to SQLite in WAL mode with audit logs.

How does the Swiss matchmaker reduce the number of comparisons?

Instead of comparing all 50,000,000 pairs in a 10,000-item set, it pairs items with adjacent Elo ratings that have not yet faced each other, turning the tournament into roughly 70,000 matches at about one bit of entropy reduction per match.

What rating engines and judge backends can I use?

Rating engines are Elo (1v1 Swiss) and TrueSkill/OpenSkill multi-player Bayesian. Judge backends are Jev, local open-weight models via Ollama, vLLM or LM Studio, an ensemble committee with majority or Borda aggregation, and a deterministic mock.

How do I run a tournament from the terminal?

python3 main.py with flags such as --mode trueskill --cohort-size 3 --judge jev and --dataset, or --concurrency 16 for a 10,000-poem Elo run. The web dashboard instead starts with python3 server.py on http://localhost:8000.

CommunityJev Tools & CLIs

AI Elo Ranker

Tournament engine that ranks texts by pairwise or cohort duels judged with Jev or local open-weight models, using Elo and TrueSkill ratings with a live web dashboard.

See highlights

Open github.com

Stars: 7
Language: Python
Published by: Community
Author: opaielsheikh
Last verified: Oct 2026

opaielsheikh/ai-elo-ranker

User repository on GitHub

View on GitHub

High-speed recursive AI Elo tournament engine powered by Jev and Swiss matchmaking

Forks: 2
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

A blazing-fast, recursive tournament engine for ranking texts (poems, startup pitches, rap lyrics, cold emails, ad hooks) using Jev LLM, local open-weight models (Qwen 2.5, Gemma 2, OpenJev, Laya), TrueSkill / OpenSkill multi-player Bayesian ratings, standard Elo mechanics, and real-time WebSocket streaming. Designed for high-throughput concurrency, cinematic live demos on X (Twitter), and web-based interactive exploration. $O(N \log N)$ Swiss-System Matchmaker vs $O(N^2)$ Round-Robin:

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import awesomejev

Works forJev for Classification

Used inJev for Software & DevTools

Languages: Python

Highlights

Judges pairwise duels and multi-candidate cohorts with Jev, local models, ensembles, or a mock.
Pairs items with a Swiss matchmaker and updates Elo or TrueSkill Bayesian ratings per outcome.
Streams duels, verdicts, leaderboards, and throughput metrics to a real-time web dashboard.
Ships CLI and server entry points, SQLite persistence, tests, and bundled text datasets.

Watch out

Needs a TypeSafe API key for the Jev judge path, plus Python environment setup for the server and CLI. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jevclip

Video tool where Jev scores subtitle segments and code assembles a short highlights cut plus a de-watered full cut with timestamped summaries.