Article
CommunityJev Models & Reimplementations
RoboJev: An Open Vision-Language Judge for Robot Success Detection
Open-weight vision-language judge that takes a task instruction plus 10 frames of a robot manipulation attempt and predicts success or failure with calibrated probabilities in a single forward pass. Released with inference code and a 20,628-prediction evaluation set.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Trained as a rank-32 LoRA adapter on JEV-27B-VL's language model, with the vision encoder kept frozen.
- Trained on 52,598 episode records and uses a source-specific KL penalty applied only to research data.
- Attains 95.2% balanced accuracy and 0.993 AUROC on 268 held-out episodes of the SO-101 robot.
- AUROC 0.890 over 459 episodes from unseen labs and robots; 93.3% accuracy on 135 community-collected episodes.
- Median 0.26 seconds per episode on a single H100, about $0.19 per 1,000 predictions at $2.69 per GPU-hour.
Questions about this entry
Short answers drawn from the linked project or page.
It takes a task instruction, 10 frames of a manipulation attempt, and, when available, final frames from auxiliary cameras, then predicts success or failure through a calibrated two-token readout that outputs probabilities in a single forward pass.
Inference runs on a single H100 at a median of 0.26 seconds per episode in serial mode, which works out to roughly $0.19 per 1,000 success predictions at $2.69 per GPU-hour, not counting media processing and serving costs.
On 268 held-out episodes collected with the same robot model, SO-101, RoboJev-27B reaches 95.2% balanced accuracy and 0.993 AUROC.
It reports AUROC 0.890 over 459 episodes from unseen labs and robots, and 93.3% accuracy over 135 clearly labelled community-collected episodes. The base model reaches AUROC 0.900 on the unseen-lab and robot subset with the same inputs.
The release includes inference code and a transparent evaluation of 20,628 predictions organized into 29 model and protocol entries, provided for browsing and reproducible benchmarking of success detection.
It is a variant of JEV-27B-VL trained with a rank-32 LoRA adapter on the language model only, keeping the vision encoder frozen, using 52,598 episode records and a source-specific KL penalty that applies only to research data.
No. The API-based frontier models received fewer input frames, 4 images from the primary camera only, while RoboJev-27B was given more frames at inference, so the comparison is not strictly like-for-like.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Standard One SH (StandardThinking)
LinkedIn: What multimodal decision models actually change, and what they do not.
LinkedInJevDirectory.org
Hacker News: Decision Index 0.2.1 scores open reproductions of Jev on a frozen suite.
jackbravo4 pointsDecision Index 0.2.1 – 70 open reproductions of TypeSafe Jev's Decision ModelRead the thread on Hacker NewsHacker News: Pointer to a DiffusionGemma plus vLLM tutorial for a Jev-shaped classifier.
1 pointHow to build a Jev-style classifier with DiffusionGemma and vLLMRead the thread on Hacker NewsMore Jev resources like this
Apache-2.0 Hugging Face repository for RSI-JEV-27b, a 27B multimodal decision model based on Qwen3.8-27B that answers supplied multiple-choice questions over images, returning a selected option and probabilities for all options in one forward pass.
Jev Models & Reimplementations#open-models#benchmark#calibration
Community
arXiv paper introducing Iris, a System One design that accepts image, text, or mixed candidates while leaving Jev's decision module untouched, with order-invariance, null-probability, and candidate-reuse results on a Multimodal-Mind2Web benchmark.
Jev Models & ReimplementationsArticle#arxiv#open-models#calibration
Community
A family of 0.8B-9B ophthalmic decision models that score candidate answers without autoregressive decoding, trained on RetiJev-Data: 78,158 clinical-decision and diagnosis questions and 50,074 guideline-grading questions. RetiJev-9B answers a case in 158 ms on one GPU.
Jev Models & ReimplementationsArticle#arxiv#open-models#benchmark
Community
arXiv paper proposing JevRec, which encodes recommendation items as variable-length semantic identifiers and adds a Jev-like decision modeling component with separate heads for length prediction and content generation, evaluated on three public datasets.
Jev Models & ReimplementationsArticle#arxiv#research#benchmark
Community
Hugging Face dataset of Jev-shaped retrieval and classification decisions built from MM-BRIGHT, holding Qwen3.6-35B-A3B-FP8 state, action, and hard-negative embeddings for post-training CLM heads.
Jev Models & Reimplementations#open-models#benchmark#evaluation
Community
Hugging Face dataset that renders HoVer claim-verification items as Jev Decision Index retrieval and classification requests, with Qwen3.6-35B-A3B-FP8 state, action, and hard-negative embeddings for post-training CLM heads.
Jev Models & Reimplementations#classification#benchmark#open-models
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.