User repository on GitHub
- Forks
- 0
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Benchmarks & Evaluations
Jev Toolspace
A Python research harness evaluating Jev (TypeSafe System One, jev-1.13.0) as a relevance judge for task-level tool spaces. One API call scores every tool with a yes/no noul question per tool, and the repo benchmarks keep-all versus per-request, per-task and trace-based spaces on TGB and BFCL.
- Stars
- 1
- Language
- Python
- Reading time
- 3 min
- Published by
- Community
- Author
- xuan7zhang
- Last verified
- Oct 2026
Highlights
- One call scores every tool in a menu: Jev answers a yes/no noul question per tool and returns independent probabilities, so tool scores do not compete.
- Three settings: A judges every request online, B averages a task's build requests into one space, C judges the executor's build traces.
- Reported Qwen2.5-7B results: on TGB, Jev-C 80.7 macro accuracy vs 69.5 keep-all; on BFCL multi-turn, Jev-B 17.3 vs 11.3 keep-all and 13.8 Tool2Vec.
- The client caches calls by the hash of (model, state, questions), throttles to 600 requests per minute, retries 429/529, and JEV_MOCK=1 scores offline.
- Protocols: TGB is 10 families x (80 build / 320 eval) with a shared K budget; BFCL multi_turn_base is 4 classes x (10 build / 40 test), menus padded to 30.
Quickstart
bash
export JEV_API_KEY=...
JEV_MOCK=1 python -m jev.tgb_run --dir $TGB --tag 7b --conds A,B,C --limit-fit 3 --limit-eval 3 --traces <traces.json> --out-dir /tmp/mockWatch out
Grounded in the truncated README and GitHub metadata reviewed on 2026-10-15; the README is cut off mid-sentence in the results discussion, the lanes need the external tgb and bfcl_eval packages plus a GPU for executor traces, and the reported numbers come from the project's own runs with a Qwen2.5-7B executor.
Questions about this entry
Short answers drawn from the linked project or page.
A Python research project that tests Jev as a relevance judge for building task-level tool spaces: instead of handing an agent the whole tool menu, Jev scores every tool so only the relevant subset keeps full documentation, with the rest compressed.
Jev answers yes/no (noul) questions about a shared state and returns an independent probability for each one, so a single API call scores every tool in a menu with one question per tool — and a tool's score does not compete with the others.
A judges each request online from the request and tool descriptions; B applies the same judgment to a task's build requests and averages them into one space per task; C judges the executor's build traces — tool outputs and the answer on TGB, calls, outputs and replies on BFCL.
With a Qwen2.5-7B executor, TGB macro accuracy is 80.7 for Jev-C against 69.5 keep-all and 67.6 for a trace judge; BFCL multi-turn is 17.3 for Jev-B against 11.3 keep-all and 13.8 Tool2Vec. These are the project's own runs on its own protocol.
Export JEV_API_KEY, set PYTHONPATH to include the tgb and bfcl_eval packages plus the repository, and use the documented commands: a JEV_MOCK=1 mock run for dry runs, bash tgb/tgb_lane.sh <GPU> 7b <model dir> all for traces, Jev and evaluation in one session, then the matching python -m metrics module.
JEV_MOCK=1 returns labeled mock scores offline, covering dry runs of the TGB driver. Real scoring needs JEV_API_KEY, and full lanes need a local executor checkpoint such as Qwen2.5-7B-Instruct on a GPU, since the drivers reuse existing benchmark code rather than re-implementing evaluators.
The README excerpt ends mid-sentence in the results discussion, raw API caches are not shipped, and the numbers come from one executor model under the project's own protocol — TGB with one shared K budget per family and BFCL menus padded to 30 tools. Treat them as a research report, not a general Jev benchmark.
More Jev resources like this
A benchmark comparing BM25, embeddings, rerankers and Jev for tool search over 525 real MCP tools and 94 annotated tasks, plus an experimental Jev search engine. Jev's two-stage search found the right tool first 56% of the time.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
kachar/jev-tool-searchTypeScript
kachar/jev-tool-searchTypeScriptBenchmark harness that scores TypeSafe Jev against any OpenRouter model or local checkpoint on labelled classification data: accuracy, calibration (ECE), confidence distribution, latency and cost across five tasks with 300 items each.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
4esv/jev-evalPython
4esv/jev-evalPythonA reproducible 100-task benchmark comparing Jev with Qwen3.5 0.8B, 2B and 4B across three thinking budgets: Jev scored 77.0%, the 2B model at 2,048 tokens 82.0%, and the 4B 96.7%.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
softpudding/jev-frontier-100Python
softpudding/jev-frontier-100PythonA Python benchmark that evaluates LLMs' reasoning and instruction-following by making them play chess against a random player or the Komodo Dragon engine in multi-turn dialogs, separating chess skill from protocol durability.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
maxim-saplin/llm_chessPython
maxim-saplin/llm_chessPythonA reproducible lab measuring how several coding agents coordinate on one repository, plus Médula, a kernel that asks TypeSafe Jev before every write whether a change collides with what the other agents are doing.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
joaquinruiz/medulaPython
joaquinruiz/medulaPythonBenchmark comparing TypeSafe Jev 1.13 with the local Laya MLX typed-decision model across 1,470 synthetic items and eight tasks: accuracy, calibration, latency and throughput, with scripts, datasets, predictions and the full report in the repo.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
harrymunro/jev-laya-benchmarkPython
harrymunro/jev-laya-benchmarkPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.