Article
CommunityJev Benchmarks & Evaluations
Beyond CLIP: Calibrated Few-Shot Defect Detection With System One Decision Models
arXiv study applying Jev and Clef-Flash to manufacturing defect detection with as few as two reference images per class, including agentic refinement of text descriptions and probability calibration. Ships replication code, an R decisions package, and a no-code web app.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Tests Jev and Clef-Flash on image inspection with as few as k=2 reference images per class at inference.
- Jev reaches 0.75-1.00 test accuracy versus 0.67-0.84 for CLIP using the same references.
- Agentic refinement of guiding text descriptions improved accuracy in 6 of 8 model-application combinations.
- The calibration approach reduced the Brier score in 13 of 15 model-application configurations.
- Findings cover five manufacturing case studies, plus replication code, an R decisions package, and a public web app.
Questions about this entry
Short answers drawn from the linked project or page.
Product and process inspection in manufacturing, where cameras and vision sensors check parts for defects. The authors run five manufacturing case studies comparing Jev and Clef-Flash against CLIP baselines.
The framework is designed for k-shot use with as few as k=2 reference images per nominal and defective class at inference, which matters in high-yield or low-volume processes where defective samples are scarce.
With two reference images per class, Jev reported test accuracies of 0.75-1.00 and Clef-Flash 0.71-0.99, against 0.67-0.84 for CLIP using the same references.
It calibrates the defect probabilities produced by the decision models so the scores are more reliable for inspection decisions. The authors report Brier score reductions in 13 of the 15 model-application configurations tested.
An agentic approach automatically refines the text descriptions that guide image classification in these decision models. The authors report significant accuracy improvements across 6 of 8 model-application combinations.
Yes. The authors provide replication code, an R package called decisions for evaluating additional models, and a publicly available web app that lets practitioners inspect their own images without writing code.
It still depends on having reference images for each class and on text descriptions that can be written and refined for the task. The results come from five manufacturing case studies, so accuracy figures should not be assumed to transfer unchanged to other domains.
More Jev resources like this
Retrospective study comparing JEV, a non-generative decision model returning native option probabilities, with schema-constrained Qwen Flash on 2,048 patient-condition pairs from TrialGPT and MIMIC-III/IV under Zero Data Retention policies.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Spring Boot and Spring AI benchmark comparing Java keyword rules, Jev 1.13.0, and Claude Haiku 5.5 on warehouse goods-receipt triage, with a 200-remark synthetic dataset and a Python analysis script.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
Communitymohammedahmadi1990/jev-triage
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv study comparing Jev 1.13 with Clef, Microsoft-Decision-1, Laya, GPT-6 Luna and GPT-6 Sol on 172 rheumatology exam questions and 70 diagnostic vignettes, reporting accuracy, calibration and language effects.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper introducing GDGraph, a guideline-derived decision graph framework that compiles radiology guidelines into an executable graph and has a System One model answer localized clinical questions, reporting 90.8% accuracy across 347 cases.
Jev Benchmarks & EvaluationsArticle#arxiv#research#evaluation
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.