Skip to content
JevDirectory.orgTraffic count unavailable. View site traffic
Ecosystem

Leaderboard

A public ranking of decision models on a shared task set, where the selection rules decide what the numbers mean.

All 164 terms
Category
Ecosystem
Also known as
—
Related terms
4
Directory entries
55
Docs
—
Added
2026-10-10

Definition

Two boards now rank this ecosystem. The Typed Decision Leaderboard scores 2,018 verifier items with the same labels for every entrant, each run on its own maker's code, and resolves ties with paired bootstrap intervals; the S1MB leaderboard spreads roughly 100 models across 100+ specialized tasks and leaves failed measurements visible.

Read provenance before rank. Boards mix independent re-runs with vendor snapshots, fine-tuned checkpoints with zero-shot baselines, and private test text with published labels, so a rank without its axis and its split is not comparable to the next board's rank.

Tagsevaluationbenchmarks

Related terms

Definitions that connect to this one.

Benchmark claimsTypeSafe's headline multipliers — 193.6x faster, 444.6x cheaper, a 238x lower input price — come from its own workflow comparisons.Evaluation & TrainingWorkflow evalsTypeSafe's published evaluations of four automation workflows, comparing Jev and frontier LLMs as structured workflows versus single prompts.Evaluation & TrainingOpen weightsModel checkpoints published for anyone to download, run, and fine-tune, as opposed to a hosted closed API like Jev.EcosystemBrier scoreA score for a full probability vector against one observed label. Lower is better, and a confident wrong answer costs more than a hedge.Evaluation & Training

From the Jev directory

Typed Decision Leaderboard
An independent 2,018-item answer-verifier benchmark: the same items and labels for every entrant, each run on its own maker's code, with Jev, ZTC-Judge, Darwin, and open models ranked by weighted AUC.
Jev Benchmarks & EvaluationsPlayground#community#evaluation#benchmarks
Community
S1MB Leaderboard
The System One Mosaic Benchmark leaderboard: about 100 decision models compared across 100+ specialized tasks, with results regenerated from a public dataset rather than estimated.
Jev Benchmarks & EvaluationsPlayground#community#evaluation#benchmarks
Community
Typed Decisions Benchmark
The 400-case, 2,000-decision benchmark open models train and report against: four workflows, state plus typed questions, and gold distributions averaged from three teacher samples.
Jev Benchmarks & EvaluationsDocs#community#benchmarks#evaluation
Community
Best Decision Model APIs in 2026
A ranked comparison of thirteen decision-model APIs as of October 2026, including Jev, OpenAI Decisions, Clef, Liquid d1, Kev, Perplexity Decider, and Strands, scored on evidence, usage, and price.
Jev Benchmarks & EvaluationsArticle#community#comparison#decision-model
Community
A decision model d1 surpasses Jev
GIGAZINE's report on Liquid AI's d1, which it describes as the first decision model to beat Jev on Hugging Face's Decision Index, and a short survey of the clone wave that followed Jev's September launch.
Jev Media & TalksArticle#community#news#benchmarks
Community
Lies, Damned Lies, and Benchmarks
TypeSafe's 11 September 2026 essay against benchmaxxing: a public eval can select the model even when nobody trains on that test.
Jev Guides & ArticlesArticle#official#benchmarks#evaluation
Official

49 more matching entries in the full directory.

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai