Article
CommunityJev Practices & Patterns
IJev: Order-Invariant Decisions via Iterated Cyclic Reordering
arXiv paper proposing IJev, an inference-time method that repeatedly reorders options by a cyclic-averaged, label-prior-corrected distribution and runs another cyclic average. It reports zero flip rate across five benchmarks and five models.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- IJev repeatedly reorders options by the cyclic-averaged, label-prior-corrected distribution and runs another cyclic average.
- Every IJev iterate is exactly input-order-invariant.
- On five benchmarks and five models, IJev has flip rate zero in every cell.
- The default uniform average over T=20 rounds substantially improves calibration over a single cyclic average, with comparable accuracy.
- A descriptive choice of horizon and averaging rule is best or tied-best for accuracy in all 25 settings.
Questions about this entry
Short answers drawn from the linked project or page.
It is any fixed function that returns the same ordered option list for every input order of a fixed option set. Starting from such an order and averaging a listwise readout over its K cyclic shifts removes the input order.
The K cyclic shifts are only one of the (K-1)! cyclic classes, and different invariant operators can select different classes. The decision therefore still depends on the chosen cyclic class.
When the iteration does not converge, IJev averages the round distributions rather than returning the last round. Every iterate remains exactly input-order-invariant.
The default is a uniform average over T=20 rounds. The paper reports this substantially improves calibration over a single cyclic average, with comparable accuracy.
On five benchmarks and five models, IJev has flip rate zero in every cell. The paper reports this as a complete result across all evaluated settings.
System One decision model probabilities carry a prior over answer labels. IJev corrects for that label prior inside the distribution used to reorder options before each cyclic average.
Its accuracy is comparable to a single cyclic average. A descriptive choice of horizon and averaging rule is best or tied-best for accuracy in all 25 settings.
More Jev resources like this
4GitHub stars
An analysis of ordinal-scale bias in JEV 1.13 and three open KEV models: across 36 ordinal datasets, final decisions use only 67-76% of the effective gold support versus 87-102% on four nominal tasks. BA-LoRA post-training raises utilization from about 47% to 86%.
Jev Practices & PatternsArticle#community#research#arxiv
Community
Glax147/jev_ordinal_scale_biaPython
Glax147/jev_ordinal_scale_biaPythonA bilingual field map of where Jev fits and where it fails, separating the author's raw API suites from cited third-party results and TypeSafe's own claims.
Jev Practices & Patterns#community#python#evaluation
Community
Zaious/jev-capability-atlasPython
Zaious/jev-capability-atlasPythonCase study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Open-weight vision-language judge that takes a task instruction plus 10 frames of a robot manipulation attempt and predicts success or failure with calibrated probabilities in a single forward pass. Released with inference code and a 20,628-prediction evaluation set.
Jev Models & ReimplementationsArticle#arxiv#open-models#evaluation
Community
arXiv study comparing Jev 1.13 with Clef, Microsoft-Decision-1, Laya, GPT-6 Luna and GPT-6 Sol on 172 rheumatology exam questions and 70 diagnostic vignettes, reporting accuracy, calibration and language effects.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.