tryaksh/jev-pick-and-place-study
User repository on GitHub
A small reproducible MuJoCo pilot comparing Jev, Claude Haiku, and reactive rules for pick-and-place.
- Forks
- 0
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
CommunityJev Practices & Patterns
Jev Pick And Place Study
A MuJoCo pick-and-place pilot that runs Jev, Claude Haiku and hand-written reactive rules through the same episodes and publishes the comparison, including the parts that did not flatter Jev.
- Stars
- 1
- Language
- Python
- Published by
- Community
- Author
- tryaksh
- Last verified
- Oct 2026
About this project
A small, reproducible MuJoCo pilot exploring whether Jev can choose the actions for a pick-and-place task, and how that compares with Claude Haiku and simple rules. The aim is to support an honest research discussion and build on it later. Watch the demo · Results table · Full results · Evaluation traces. Ten unseen starting seeds, each tested normally and with a target change after lifting: 20 episodes per controller. Same observations, small actions, motion controller, success test, and 100-decision limit.
Quoted from the project's README. Read the full README.
Used inJev for Software & DevTools
- Languages
- Python
Highlights
- Rules and Jev each cleared 10 of 10 ordinary and 10 of 10 changed-target episodes; Haiku managed 4 and 1.
- Jev chose the same action sequence as the rules controller in all 20 episodes, with no task-performance advantage from AI.
- Median decision time was 146 ms for Jev against 732 ms for Haiku, with API cost quoted per successful run.
- The demo replays from a bundled trace archive with no API key: unzip evaluation_traces.zip, then run demo.py render.
Watch out
The README is direct about the limits of its own result: this is one engineered interface and prompt setting, not a general model ranking. Haiku ran with structured action-only output and thinking disabled. The shared motion controller already follows the current target, so the changed-target result is not evidence of sophisticated AI replanning. Success requires a grasped lift, transfer, release, retraction and stable placement, and there is no vision or training involved. Twenty trials are exploratory evidence, and completion estimates add measured decision time to simulated duration, so they are not hardware cycle times. The commands are PowerShell for Windows with Python 3.11 and uv. Paid runs are capped at $1 for Jev and $5 for Claude, and the per-evaluation API spend quoted is $0.0695 for Jev against $2.0870 for Claude. Copy reviewed against the repository README on 2026-10-04, not from running it.
Questions about this entry
Short answers drawn from the linked project or page.
It tied them and cost more. The main finding as stated is that Jev chose exactly the same action sequence as the rules controller in all 20 episodes: 10 of 10 on ordinary seeds and 10 of 10 with a target change after lifting, against 4 of 10 and 1 of 10 for Claude Haiku 4.5. The author's read is that the rules are under 0.01 ms and free, Jev ran 146 ms median per decision, so this fixed task showed no task-performance advantage from adding AI.
Not much, and the README says so in those terms: the shared controller already follows the current target, so the changed-target result is not evidence of sophisticated AI replanning. Jev chooses one bounded command every 0.2 simulated seconds and shared code handles the geometry, executing movements of 5-20 mm from exact positions, contact and alignment flags plus the explicit task rules.
Budget for the Claude arm. The quoted figures are $0.0035 per success for Jev against $0.4174 for Haiku, with evaluation API spend of $0.0695 for Jev and $2.0870 for Claude, and the prices are estimates from returned token usage. The README states hard caps of $1 for Jev and $5 for Claude, and recommends keeping the local spending ledgers and traces to stay under them and resume progress.
Yes. The bundled results/evaluation_traces.zip holds the recorded turns, and demo.py render replays them with no API calls and no keys. Everything else needs uv, Python 3.11, a .env with your own keys and Windows commands; note that .env is git-ignored and the README says never to commit it.
Success requires a grasped lift, transfer, release, retraction and stable placement. The model chooses one bounded command every 0.2 simulated seconds, shared code handles geometry and executes 5-20 mm movements, and there is a 100-decision limit; the simulation pauses during API calls, so completion estimates are not hardware cycle times.
The demo uses smoother motion increments and actual 500 Hz physics states exported at 50 fps, with playback speed and omitted API waits labelled. The yellow disk is a flat tabletop target marker, and the protocol and raw traces are included in the repository.
Replace the method in compare.py --method rules --split eval --start 1001 --count 10 with jev or claude for paid runs. Caps are $1 for Jev and $5 for Claude, so keep the local spending ledgers and traces to stay under them and resume progress; credentials and generated working files are Git-ignored.
More Jev resources like this
2GitHub stars
Runs Jev as an LLM router over two public benchmarks, with a no-Jev ablation and RouterArena's own scorer so the numbers sit next to the public leaderboard.
Jev Practices & Patterns#community#jevbest#catalog-import
24GitHub stars
Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
rorshopping/jev-on-a-laptopPython
rorshopping/jev-on-a-laptopPython22GitHub stars
Offline evaluator that checks whether classifier confidences match reality and converts mistake costs into a deployable threshold, emitting an HTML report plus machine-readable threshold files.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
rlaope/jevalPython
rlaope/jevalPython16GitHub stars
Vendor-neutral harness that tests whether Jev reranking improves a small RAG stack, using frozen hybrid BM25 plus BGE-M3 candidates and isolating retrieval, rerank, and generation stages.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
erendikmenn/jev-rag-benchmarkPython
erendikmenn/jev-rag-benchmarkPython37GitHub stars
Minimal Python recipe that turns any causal LLM into a Jev-style classifier by prefilling choice_index and constraining sampling to legal indexes in one forward pass.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
sgoedecke/system-onePython
sgoedecke/system-onePython39GitHub stars
Open replica of the typed decision API on openly available base models, answering choice, score, and true-probability questions in one forward pass with no decoding.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
mithalouni/system-one-openPython
mithalouni/system-one-openPythonFrom the Jev guides
Original write-ups on what this entry is for.
Most Jev accuracy problems are state problems. What belongs in a state, what is a pre-made judgment to leave out, and why naming the slice you ask about matters more than the wording.
Jev Practices & Patterns#state#design#primitives
Read article
A Noul returns one number and no separate confidence. How to read the value, pin its boundary with criteria, split compound questions, and set a per-action threshold that matches the cost of being wrong.
Jev Practices & Patterns#noul#primitives#confidence
Read article
A Score is a weighted position, not a ruler. Why the number can land between levels, why levels are judged separately, and how to combine several atomic scores in code instead of one crowded rubric.
Jev Practices & Patterns#score#primitives#scoring
Read article
From the Jev community
Posts from builders shipping with Jev right now.