User repository on GitHub
- Forks
- 13
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Practices & Patterns
Jev Eval Agent
A personal-assistant agent with 100 mocked tools that measures how many steps a task takes when the LLM picks the tool versus when Jev picks it before every model step.
- Stars
- 108
- Language
- HTML
- Reading time
- 3 min
- Published by
- Community
- Author
- vinilana
- Last verified
- Sep 2026
About this project
🇺🇸 English · 🇧🇷 Leia em português. A personal-assistant agent built with eve (Vercel), with 100 mocked tools, served through OpenRouter. The repository exists to answer one question: how many steps does the agent need to finish the same task when the LLM picks the tool itself vs. when Jev (TypeSafe's classifier) picks it? Everything lives on main; the mode is a single environment variable. Instructions, catalog, evals and UI are identical in both modes.
Quoted from the project's README. Read the full README.
Highlights
- AGENT_MODE switches between llm-direct, where the LLM sees all 100 tools each step, and jev-classifier, where Jev picks before every model step.
- Confidence-gated routing: if Jev picks respond_to_user but done is below JEV_DONE_THRESHOLD (0.5), the reply is blocked and the next-best tool is exposed.
- Six tasks (three basic, three complex) include look-alike distractor tools; calling one counts as a wrong tool in the report.
- Per-run metrics include modelSteps, toolCalls, distractorCalls, completion, efficiency, token usage, and estimated OpenRouter cost.
Quickstart
bash
cp .env.example .env.local # OPENROUTER_API_KEY and TYPESAFE_API_KEY
npm install
EVAL_MODEL=deepseek-v4.1-flash npm run evalWatch out
No license file, so reuse terms are unclear. Needs Node.js, an OPENROUTER_API_KEY and a TYPESAFE_API_KEY, and the benchmark prompts are in Portuguese.
Questions about this entry
Short answers drawn from the linked project or page.
eval-results/ is versioned, so the dashboard and deck open with the numbers already in the repo, and each run overwrites <mode>/<model>/<task>.json and appends to history.jsonl. Nothing records who ran them or when, so treat them as a starting point rather than a published evaluation.
Who selects the tool. In llm-direct, the default, the LLM sees all 100 tools on every step. In jev-classifier, Jev is called before every model step with the conversation state and only the tool it picked is exposed, so the model just fills in arguments. Instructions, catalog, evals and UI are identical; one environment variable switches it.
No, they are mocked: 100 tools with a name, description, zod schema and mock executor, running against a deterministic mock world of calendar, contacts, flights and expenses. That is what makes step counts comparable across modes, and it also means the numbers say nothing about real tool reliability.
Yes. AGENT_MODEL=mock runs llm-direct, and adding AGENT_MODE=jev-classifier with JEV_STUB=1 runs the other path; the README says this only validates the pipeline. Real runs need Node 24+, an OPENROUTER_API_KEY and a TYPESAFE_API_KEY.
Requires Node 24+. Copy `.env.example` to `.env.local` and fill in OPENROUTER_API_KEY and TYPESAFE_API_KEY, run `npm install`, then `npm run eval` for one mode or `npm run eval:all` for modes times models. Results land in eval-results/<mode>/<model>/<task>.json.
Jev is asked two questions about the same state each step: next_tool, a choice over the 100 tools plus respond_to_user, and done, a noul. If Jev picks respond_to_user but done is below JEV_DONE_THRESHOLD (default 0.5), the router blocks the reply and exposes the next-best tool instead.
modelSteps, toolCalls, extraToolCalls, distractorCalls, jevCalls, completion, idealPath, efficiency, usage with tokens and estimatedCostUsd, reasoningChars and durationMs, plus a step-by-step timeline. Jev's decisions also go to eval-results/jev-trace.jsonl with confidence and top-5.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Tool calling as classify plus action, back to twelve-factor agents
X: A chat bot with no LLM
More Jev resources like this
An independent calibration study of Jev against a frontier fallback on Banking77 and Web of Science, packaged as the janus-decide tool that measures a routing threshold on your own data instead of shipping a default one.
Jev Practices & Patterns#community#python#routing
Community
FirasSX914/JanusPython
FirasSX914/JanusPythonSixteen thousand calls against gpt-5.4-mini and gpt-5.6-luna on four public datasets plus production pipeline decisions, with a threshold procedure and an explicit filter-not-replacement verdict.
Jev Practices & PatternsArticle#community#benchmarks#evaluation
Community
A comparison arena that runs the same batch of review comments through Jev and DeepSeek, showing processing time, cost, and per-label results with CSV/Excel import, replay, and offline reports.
Jev Practices & Patterns#community#javascript#benchmarks
Community
NanmiCoder/jev-arenaJavaScript
NanmiCoder/jev-arenaJavaScriptSide-by-side benchmark of TypeSafe Jev, Qwen 3.8 27B on Cerebras, and a local Needle 3 across seven synthetic workloads, recording validated outputs, mistakes, latency, tokens, and estimated cost. Raw exports and per-scene limitations are published.
Jev Practices & Patterns#community#typescript#benchmarks
A probability-aware evaluation harness that compares TypeSafe Jev with GLiNER2.5 on zero-shot single-label text classification, measuring calibration, coverage at a fixed error budget, latency, and token cost.
Jev Practices & Patterns#community#python#benchmarks
Community
AbdelStark/jev-benchmarksPython
AbdelStark/jev-benchmarksPythonOpen spike issue in the Orcha multi-agent orchestration platform: it proposes moving judgment out of expensive state-of-the-art models and onto Jev, which emits precise instructions for a cheaper executor to carry out, and requires findings, a cost model, and a recommendation as issue comments.
Jev Practices & Patterns#community#orchestration#cost-control
Community
open-orcha/orchaTypeScript
open-orcha/orchaTypeScriptFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.