Article
CommunityJev Benchmarks & Evaluations
Do Multimodal Models Decide Consistently on Equivalent Optical Spectra?
arXiv paper testing representation invariance and abstention on measured optical spectra of silicon metasurfaces and gold nanoparticles, evaluating d3-nano, Jev 1.13, three GPT-6 Luna configurations, and Qwen3.5-2B.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Each spectrum is shown as four tables (nm, µm, scientific notation, eV) and two plots (nm, eV).
- Changing only the representation moves accuracy by up to 41 percentage points within one model.
- Asking in photon energy instead of wavelength helps some model-view pairs by 16-50 points and hurts others by up to 25.
- LoRA fine-tuning of d3-nano abstains whenever input is truncated; training with answerable truncations avoids that shortcut in all six seeds.
- The tasks, per-response results, and analysis code are released.
Questions about this entry
Short answers drawn from the linked project or page.
The decision models d3-nano and Jev 1.13, three GPT-6 Luna configurations, and the Qwen3.5-2B base model, tested on measured optical spectra of silicon metasurfaces (MOCLIP) and gold nanoparticles with labels computed from the data.
The same spectrum is presented as four tables (nm, µm, scientific notation, eV) and two plots (nm, eV), and the decision should not depend on which representation the model sees.
Changing only the representation moves accuracy by up to 41 percentage points within one model, mostly on photon-energy tables and plots.
Partial spectra remove data inside or outside the queried range. When the relevant extremum is removed, unadapted models rarely answer "undetermined".
Much of its gain on unanswerable questions came from abstaining whenever the input is truncated. Training with answerable truncations avoided the shortcut in all six seeds.
A 90-100 point gap between abstention after inside and outside cuts on held-out structures. With the same amount of extra original-task data, five of six seeds kept the shortcut.
Yes. The authors release the tasks, the per-response results, and the analysis code.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Standard One SH (StandardThinking)
LinkedIn: What multimodal decision models actually change, and what they do not.
LinkedInJevDirectory.org
Hacker News: Decision Index 0.2.1 scores open reproductions of Jev on a frozen suite.
jackbravo4 pointsDecision Index 0.2.1 – 70 open reproductions of TypeSafe Jev's Decision ModelRead the thread on Hacker NewsHacker News: Pointer to a DiffusionGemma plus vLLM tutorial for a Jev-shaped classifier.
1 pointHow to build a Jev-style classifier with DiffusionGemma and vLLMRead the thread on Hacker NewsMore Jev resources like this
Benchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
Controlled audit of presentation sensitivity in typed decision models, comparing Jev, Clef Flash, and Perplexity Decider across native choice and score interfaces on 9,000 multiply annotated humor, helpfulness, and summarization items.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper introducing S1CogBench, a benchmark that runs 24 cognitive experiments against System One decision models, including Jev and five open models, to test whether typed decisions respond to evidence, feedback, memory load, and future opportunities.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper presenting a bias audit protocol for System One decision models, covering BBQ, CrowS-Pairs, and SilverSpoon adapted to the typed interface, twelve models plus a no-thinking ablation, and 1,638,000 decisions with swapped options.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv evaluation of 14 System-1 decision models, including Jev, against four generative judges on eleven judge benchmarks, covering accuracy, calibration, cost, and robustness to instructions injected into judged responses.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.