Article
CommunityJev Benchmarks & Evaluations
S²-MAS: Society-Scale Multi-Agent Simulation Framework
arXiv paper introducing S²-MAS, a society-scale multi-agent simulation framework built on 500 million synthetic household personas from survey data. Its Genese engine separates decisions from state updates, and sharp-421m-mlx is benchmarked against Jev 1.13 on financial-choice tasks.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Oct 2026
Highlights
- Framework built on 500 million synthetic household personas grounded in real-survey data
- Genese engine separates decisions from state updates
- sharp-421m-mlx trained with SFT + RLCD on 2.93 million examples
- Reaches 74.55% accuracy and 38.37% question-macro F1 on one-shot consumer financial-choice tasks
- Evaluates eight financial task families and simulates 12 million distinct household agents
Questions about this entry
Short answers drawn from the linked project or page.
S²-MAS is a society-scale multi-agent simulation framework built on 500 million synthetic household personas grounded in real-survey data. Its Genese engine separates decisions from state updates, while reusable financial scenarios and application ledgers track consumption, cash, debt and payments across simulated households.
sharp-421m-mlx achieves 74.55% accuracy and 38.37% question-macro F1 on one-shot consumer financial-choice tasks derived from held-out survey responses.
Its accuracy exceeds GLM-5.3-Flash, Jev 1.13, DeepSeek-V4.1-Flash, GPT-6-Luna and GPT-6-Astra in the common benchmark. The full test set is used for the first four models and a matched 500-group sample for GPT-6-Astra.
The model was trained with SFT + RLCD on 2.93 million examples, then evaluated on one-shot consumer financial-choice tasks derived from held-out survey responses.
The Genese engine separates decisions from state updates, so household decisions are handled apart from the updating of state. Reusable financial scenarios and application ledgers then track consumption, cash, debt and payments over the simulation.
Jointly raising purchase goals and approved credit limits increases consumption, with responses differing across age and income groups. Education, employment and household-size differences persist after age-income standardization and mainly reflect transitions from deferral to purchase.
The comparison is not identical across all models: four models are run on the full test set while GPT-6-Astra is evaluated on a matched 500-group sample. Asset and debt groups also differ in cash drawdown, repayment and closing debt.
More Jev resources like this
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper scoring candidate pages by the citation share they will receive, comparing pretrained rerankers, Jev, eight open decision models and distilled students on GEO-Bench queries answered by ten engines. Releases GEO Citation Ranker 4B, the benchmark, all scorer predictions and code.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv study evaluating Jev as a decision reader of truncated LLM reasoning across seven reasoners and five multiple-choice benchmarks, comparing accuracy and estimated cost against forced answering at budgets from 500 to 16,000 tokens.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv study comparing Jev 1.13 with Clef, Microsoft-Decision-1, Laya, GPT-6 Luna and GPT-6 Sol on 172 rheumatology exam questions and 70 diagnostic vignettes, reporting accuracy, calibration and language effects.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper measuring gender bias in decision models, testing TypeSafe's closed Jev and five open models on gender-swapped yes/no decisions, BBQ, StereoSet, and Fulgu and Capraro's gender-bias studies, with results on top-answer flips and confidence-gate effects.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.