Jev directory
CommunityJev Benchmarks & Evaluations
Hello ZIO TypeSafe AI
Scala 3 benchmark running one Jackson documentation-research task over an MCP catalog through three designs: a direct LLM agent, an LLM with a nested Jev evidence tool, and top-level Jev orchestration, comparing turns, MCP calls, token use and latency.
- Stars
- 2
- Reading time
- 2 min
- Published by
- Community
- Added
- Oct 2026
Highlights
- Scala 3 benchmark compares a direct LLM agent, a nested Jev evidence tool, and top-level Jev orchestration.
- Runs one Jackson documentation-research task against the live javadocs.dev MCP catalog.
- Seven unique variants completed in a live sample run on 2026-09-23 using Jev 1.13.0.
- Measures controller turns, MCP calls, LLM and Jev token use, and end-to-end wall time.
- Experiments require an AWS Bedrock bearer token and a TypeSafe API key.
Questions about this entry
Short answers drawn from the linked project or page.
Set AWS_BEARER_TOKEN_BEDROCK and BEDROCK_MODEL_ID, run ./sbta "run llm-loop", then set TYPESAFE_API_KEY and run variants such as ./sbta "run jev-loop plan".
The sample uses Bedrock model ID us.anthropic.claude-sonnet-4-5-20250929-v1:0 and Jev 1.13.0 against the live javadocs.dev MCP catalog.
It compares a direct LLM-agent baseline, an LLM with a nested Jev evidence tool, and top-level Jev orchestration with plan and no-plan modes.
Runs contact Amazon Bedrock for model calls, TypeSafe AI's Jev controller via a TypeSafe API key, and the live javadocs.dev MCP catalog for documentation research.
No. The digest describes them as stochastic, network-dependent single samples, not statistically significant rankings, and recommends multiple samples plus answer-quality judging in zio-evals.
It is a Scala 3 application using ZIO, Amazon Bedrock transport, TypeSafe AI's Jev controller, and MCP wiring; experiments require a Bedrock bearer token and a TypeSafe API key.
Avoid drawing release or architecture conclusions from its single sample; the digest says to use multiple samples and answer-quality judging in zio-evals before such decisions.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Jev in front of the HeyGen MCP
X: Playwright CLI + Jev vs Playwright MCP
X: A local MCP server that cuts computer use from 23s to 0.8s. Cartwright built Cortex, a local MCP server where…
X: A side panel that drives any site's WebMCP tools
More Jev resources like this
Evaluation scripts, MCP tools, and a skill router built to measure where jev-1.13.0 fits in agent harnesses, with raw results under eval/.
Jev Benchmarks & Evaluations#community#evaluation#agents
Community
Aitejiu/jev-harness-labPython
Aitejiu/jev-harness-labPythonThe benchmark behind JevAgentHarness's paper on decision models that share control with LLMs: next-tool decisions scored at isolated, open-loop and closed-loop levels, with 3,939 common-state questions and 153,621 answers from 39 models. The L1 question texts are third-party data and are not redistributed. CC BY 4.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#agents
Community
Twelve million canonical agent-decision records built from public agent trajectories, canonicalizing state plus candidate decisions into target, outcome and eligibility supervision under CC BY 4.0.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
An open Claude Cowork-style workspace whose eval testkit can call Jev as a typed verification judge on agent-produced artifacts.
Jev Benchmarks & Evaluations#community#typescript#agents
Community
different-ai/openworkTypeScript
different-ai/openworkTypeScriptNotebook benchmark that compares Jev with open models and specialised tools on five document-pipeline chores: language detection (Choice(12)), page orientation (Choice(4)), RVL-CDIP classification (Choice(16)), bundle splitting and parse-tier triage, with liteparse doing all PDF rendering and OCR work.
Jev Benchmarks & Evaluations#community#evaluation#document-processing
Community
run-llama/jev_vs_ossJupyter Notebook
run-llama/jev_vs_ossJupyter NotebookBenchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.