Organization repository on GitHub
- Forks
- 11
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.Homepage
CommunityJev Benchmarks & Evaluations
Convex Evals
Convex's own coding evals, plus a separate multiple-choice decision benchmark in the same repo. Jev is wired into the decision side through OpenRouter's alpha decisions endpoint with a pinned version, while the coding evals still route as chat completions.
- Stars
- 130
- Language
- TypeScript
- License
- APACHE-2.0
- Published by
- Community
- Author
- get-convex
- Last verified
- Oct 2026
About this project
Convex is an open-source, reactive database that's the best platform for full-stack AI coding. We ensure that Convex performs well with a large set of models by continuously running evals. Each eval has a set prompts for coding a Convex backend, a set of human-curated solutions, and a script for evaluating the LLM's output. These evals are split up into eight different categories: The most up to date eval runs can be found on our website.
Quoted from the project's README. Read the full README.
- Languages
- TypeScript
Highlights
- The decision bank is 108 questions over 90 of the 112 coding evals, with the 22 omissions recorded in decision-bank.json and averaging done per source eval.
- Jev calls go to OpenRouter's /api/alpha/decisions with native typed questions; language models use /api/v1/chat/completions, so the two are not alike.
- A local trial needs one OpenRouter key: bun run decisions validate, then dry-run, then run, writing to output-decisions/ without reporting to Convex.
- Every question has archived executable evidence with hash checks, but only five have clean-checkout replay commands; the other 103 still need portability work.
Watch out
The repo is careful about what its scores mean, and you should read that before quoting them: a correct multiple-choice answer does not establish that a model can write or debug a working Convex application, and a miss on an unprompted-selection eval does not mean an alternative implementation was broken. Invalid responses and provider failures count as incorrect and interrupted or filtered runs are excluded, which shapes every number. Unknown provider cost stays unknown rather than being estimated. Reproducibility is uneven, since 103 of 108 questions lack supported replay commands even though the decision-verification.tgz asset carries the same evidence, locks and scripts. Local runs also need a Convex deployment for codegen, and a pass over every curated model is priced in the README at roughly 87 dollars, so pass a model explicitly. Copy reviewed against the repository README on 2026-10-04, not from running it.
Questions about this entry
Short answers drawn from the linked project or page.
No, and the README says so in the decision section itself: the multiple-choice format has its own benchmark version and its own leaderboard, and a correct choice does not establish that a model can write or debug a working Convex application. Read it as a knowledge and selection signal only. The coding evals, which deploy and grade real code, are a separate track and a separate score.
Mostly. Install bun, put an OpenRouter key in .env, run bun run setup, then the decisions validate, dry-run and run commands give you a small trial, and those commands write reports to output-decisions/ without reporting to Convex. Direct TypeSafe trials work with --provider typesafe, --model jev-latest and TYPESAFE_API_KEY, kept as a separate provider identity. The catch is scale: a pass over every curated model is around 87 dollars, so use --limit-evals and an explicit --max-known-cost-usd.
Because it is not a chat model. It is called through OpenRouter's alpha decisions endpoint with native typed questions, so the provider returns a chosen option with probabilities and confidence; the language models use standard chat completions, and the hosted config sends no chat reasoning or output-token settings for the decision model. That asymmetry is the main thing to keep in mind when reading the two sets of numbers together. The hosted config also pins a specific Jev version and records the model id the provider actually returned.
There is real process: a question authoring standard and verification guide you are told to read before changing anything, an accepted bank with documented intentional omissions, three matched option permutations in the hosted workflow, and per-source-eval averaging so one large eval cannot dominate. Against that, 103 of 108 questions still lack supported clean-checkout replay commands, so the evidence is archived and hash-checked but not yet portable. A bank update record describes the reviewed changes.
108 questions covering 90 of the 112 coding evals, with 22 intentional omissions recorded in decision-bank.json. Questions are averaged within each source eval, then source evals receive equal weight, and invalid responses or provider failures count as incorrect.
With OPENROUTER_API_KEY set, run bun run decisions validate, then bun run decisions dry-run --provider openrouter --model typesafe/jev-1.13 --limit-evals 2, then the same command with run. Local commands write reports and raw evidence under output-decisions/ without reporting to Convex.
It pins typesafe/jev-1.13, records the returned model ID, probabilities, confidence and provider-reported cost, and sends no chat reasoning or output-token settings. Every model, Jev included, goes through OpenRouter; there is no direct provider.
More Jev resources like this
4.9kGitHub stars
LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.
Jev Benchmarks & Evaluations#community#typescript#evaluation
Community
langwatch/langwatchTypeScript
langwatch/langwatchTypeScript338GitHub stars
SREGym stands up live Kubernetes clusters so SRE agents can be evaluated against 90 fault scenarios, including OS-level faults and metastable and concurrent failures, in containers that cannot see the grading logic.
Jev Benchmarks & Evaluations#community#jevbest
Community
SREGym/SREGymPython
SREGym/SREGymPython217GitHub stars
Standardizes hundreds of public datasets into interchangeable multiple-choice, classification, and token-classification templates, and deterministically recasts them into typed choice, score, and noul decision requests.
Jev Benchmarks & Evaluations#community#jevbest
Community
sileod/tasksourcePython
sileod/tasksourcePython0GitHub stars
TypeScript gateway that fans one payload out to seven Jev risk and routing questions in one call, then enforces deterministic policy verdicts.
Jev Benchmarks & Evaluations#community#github
Community
intelliDean/reflexgateTypeScript
intelliDean/reflexgateTypeScript24kGitHub stars
An open Claude Cowork-style workspace whose eval testkit can call Jev as a typed verification judge on agent-produced artifacts.
Jev Benchmarks & Evaluations#community#typescript#agents
Community
different-ai/openworkTypeScript
different-ai/openworkTypeScript197GitHub stars
A Chinese-first desktop workbench for local models whose JEV page runs typed noul, choice and score questions against a local engine or a cloud key, alongside a gateway, RAG knowledge base and shared memory.
Jev Tools & CLIs#community#jevbest
Community
kunpengtalk/OmniStudioTypeScript
kunpengtalk/OmniStudioTypeScriptFrom the Jev community
Posts from builders shipping with Jev right now.