Organization repository on GitHub
Putting TypeSafe's Jev to work on robots, fleets, and edge hardware — real measured numbers, honestly caveated.
- Forks
- 0
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.Homepage
CommunityJev Benchmarks & Evaluations
Jev for Physical AI
Measured robotics study that runs Jev as the decision layer for a 10,000-robot warehouse fleet over 41 bilingual incident templates and publishes a crossover against a self-hosted ModernBERT: 0.527 s p50, $24.57 per million decisions, and 91.3% agreement with template labels.
- Stars
- 3
- Language
- Python
- License
- MIT
- Published by
- Community
- Last verified
- Oct 2026
About this project
Real measured numbers putting TypeSafe's Jev to work on robots, fleets, and edge hardware. Jev is a "System One" model: structured state in, typed probabilistic decisions out — no text generation, 70–500ms, $0.042 per million input tokens with output free. Everyone demos it on browsers and games. We point it at the physical world instead: warehouse robot fleets, incident triage, and the build-vs-buy math of running your own models on edge CPUs.
Quoted from the project's README. Read the full README.
- Languages
- Python
- Platforms
- OpenRouter
- Capabilities
- Open sourceRequires API key
Highlights
- 300 incidents through OpenRouter typesafe/jev-1.13: 0.527 s p50, 0.813 s p95, $24.57 per million decisions
- Three judgments per call: escalate to human (Noul), owning team (Choice), urgency (Score 0-2)
- 91.3% team agreement (274/300) with template labels, stated as simulated-data agreement
- Self-hosted break-even near 977K decisions/month, about 678 robots at 48 decisions a day
- 10,000 robots x 48 decisions/day x 30 days = 14.4M decisions/month, modeled at $353.81 with Jev
Watch out
MIT code and CC-BY-4.0 benchmark data. Incidents are simulated; the README states there was no native /v1/systemone call, no calibration study and no real hardware test.
Questions about this entry
Short answers drawn from the linked project or page.
The crossover math puts break-even near 977K decisions per month on a $24/month VM, roughly 678 robots at 48 decisions a day. Below that, Jev avoids training, labeling and infrastructure work.
They come from runs executed on 2026-09-19 with stated caveats: simulated incidents, agreement measured against template labels rather than production accuracy, and a cost-only comparison against GPT-4o-mini.
41 bilingual CN/EN incident templates covering LiDAR degradation, localization drift, battery faults, pallet detection misses, network partitions and human-zone intrusions, with 300 incidents sampled. Each call asks three simultaneous judgments: escalate to human as yes or no, owning team as a choice, and urgency as a score 0-2. All 300 calls went through OpenRouter on typesafe/jev-1.13.
300 of 300 decisions succeeded at p50 0.527 s, p95 0.813 s and mean 0.558 s, with 584.9 input tokens on average. That is $0.0000246 per decision, $24.57 per million, and $0.00737 for the whole run. Team agreement with the template labels was 274 of 300, which the README calls agreement rather than accuracy.
answerdotai/ModernBERT-base at 149M parameters on a 2-core AMD EPYC, CPU-only, batch 1, with a frozen encoder, mean pooling and nearest centroid over 18 hand-written exemplars on 100 test incidents. It was faster, 169.3 ms p50 against Jev's 527 ms and 5.06 samples per second, but returned one label per call against Jev's three judgments, and agreed with Jev on 66 of 100.
Little beyond cost. It is estimated at $0.15 per million input and $0.60 per million output tokens with 600 input and 60 output tokens per call, giving $1,814.40 a month against Jev's $353.81 for 14.4M decisions, about 5.1 times. The README says the authors did not measure its triage quality or latency, so the ratio is a cost ratio, not a quality claim.
Everything under code/ is runnable: demo-a.py for the 300 real OpenRouter decisions, demo-b.py for the self-hosted comparison, plus charts.py, crossover.py and render_video.py. Demo A needs OPENROUTER_API_KEY and Demo B downloads about 599 MB of ModernBERT weights from HuggingFace. If a proxy is set, override NO_PROXY=localhost,127.0.0.1, because bare IPv6 entries crash newer httpx. Code is MIT and the measured data is CC-BY 4.0.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Hacker News: Pick Verify: a single decision at the warehouse pick station catches confusable picks.
Show HN: Verifying robot picks with a decision modelRead the thread on Hacker NewsX: A dual-arm robot with Jev in the middle layer
YouTube: Dan Makes Things drives an ESP32 robot with Jev decisions.
More Jev resources like this
2GitHub stars
Pre-registered benchmark adding Jev 1.13 to Stanford MedHELM's MedHallu results: one Noul asks whether a proposed answer misrepresents its PubMed abstract. Jev scored 92.9% on 1,000 test items at a 204 ms median and USD 0.03 per 1,000 checks through OpenRouter.
Jev Benchmarks & Evaluations#community#medical#hallucination
Community
stperic/jev-medhallu-benchmarkPython
stperic/jev-medhallu-benchmarkPython187.5kGitHub stars
Autonomous agent platform with 187,504 stars for building and running agents from plain English or a visual builder. Its platform features first-class TypeSafe Jev decision blocks for typed routing, filtering, scoring and confidence-gated next-action dispatching.
Jev Integrations & Adapters#community#autonomous-agents#platform
Community
Significant-Gravitas/AutoGPTPython
Significant-Gravitas/AutoGPTPython8.3kGitHub stars
Agentic search evaluation inside Zilliz DeepSearcher: a standalone harness replaces the ChainOfRAG reflection call with one Jev Noul per round to decide whether to stop within a seven-round budget, then compares stopping behavior, evidence recall and decision cost on 100 sampled 2WikiMultiHopQA queries.
Jev Benchmarks & Evaluations#community#rag#evaluation
Community
zilliztech/deep-searcherPython
zilliztech/deep-searcherPython21GitHub stars
MIT-licensed Python tool router for coding agents that keeps obvious calls on a fast path, asks Jev (default typesafe/jev-1.13 via OpenRouter) for genuinely ambiguous choices, and can run bounded MCTS when multi-step consequences matter. Ships a CLI, MCP server and hook integrations.
Jev Tools & CLIs#community#routing#mcp
9GitHub stars
Epistemic runtime for autonomous coding agents that uses Jev as a sub-100ms System 1 gate to shadow-evaluate capsule facts, propose evictions before bloat and prefetch memory between turns. Its fixture run reports 31.3% fewer turns and 52.6% fewer blind discovery operations, with a fail-open circuit breaker.
Jev Integrations & Adapters#community#agents#context-engineering
Community
wojciechwiesner/jit-contextPython
wojciechwiesner/jit-contextPython3GitHub stars
Evaluation scripts, MCP tools, and a skill router built to measure where jev-1.13.0 fits in agent harnesses, with raw results under eval/.
Jev Benchmarks & Evaluations#community#evaluation#agents
Community
Aitejiu/jev-harness-labPython
Aitejiu/jev-harness-labPythonFrom the Jev community
Posts from builders shipping with Jev right now.