How large is the Jev-ToM evaluation?

It compares five typed decision models and seven generative baselines on 11,117 English text questions drawn from nine Theory of Mind benchmarks, organized into six capability groups.

How accurately does Jev answer ToM questions?

TypeSafe Jev reaches 73.2% source-balanced accuracy including native controls, and ranks first or second across all six capability groups.

Which capability groups does Jev lead?

Jev leads beliefs, goals, desires and intentions, and emotions, according to the abstract, while other typed models show distinct strengths across tasks.

What drives Jev's downstream gains?

Related-question cues account for most of its downstream gains, and predicted states have task-dependent effects, the authors report.

Does confidence-based selection work for ToM?

It remains benchmark-dependent. The paper also finds supplied reasons can repair errors or overturn correct answers depending on how they are attached.

What integration priorities do the authors recommend?

The findings motivate testing Jev as a ToM selection component in assistants, with task-specific confidence calibration, candidate design, and output validation as integration priorities.

Where is the code and evaluation material?

The authors publish code and evaluation materials at github.com/HKUST-KnowComp/Jev-ToM.

CommunityJev Benchmarks & Evaluations

Jev-ToM: Evaluating Theory of Mind in Typed Decision Models (arXiv 2610.18610)

A benchmark evaluating Theory of Mind selection in typed decision models against generative baselines: 11,117 English questions from nine ToM benchmarks across six capability groups. TypeSafe Jev reaches 73.2% source-balanced accuracy and ranks first or second on every capability.

See highlights

Open arxiv.org HKUST-KnowComp/Jev-ToM

Stars: 0
Language: Python
Format: Article
Published by: Community
Last verified: Oct 2026

HKUST-KnowComp/Jev-ToM

Organization repository on GitHub

View on GitHub

Jev-ToM: Evaluating Theory of Mind in Typed Decision Models. Evaluation materials, saved predictions, and offline scoring.

Forks: 0
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.

Tagscommunity research arxiv theory-of-mind agents evaluation

Works forJev for Classification Jev for Evaluation Jev for Decision gating

Used inJev for Software & DevTools Jev for Customer Support

Languages: Python

Highlights

Compares five typed decision models with seven generative baselines on 11,117 English questions from nine ToM benchmarks.
TypeSafe Jev reaches 73.2% source-balanced accuracy including native controls.
Jev ranks first or second across all six capability groups, leading beliefs, goals and intentions, and emotions.
Related-question cues account for most of Jev's downstream gains, with task-dependent effects from predicted states.
Confidence-based selection remains benchmark-dependent, and supplied reasons can repair errors or overturn answers.

Watch out

arXiv preprint posted October 13, 2026 (cs.CL, cs.AI) by Kawai Chung and co-authors. The results cover benchmark questions rather than live assistant deployments, and the abstract reports that confidence-based selection stays benchmark-dependent and that reasons given to the model can overturn correct answers.

Questions about this entry

Short answers drawn from the linked project or page.

Related terms

Glossary definitions related to this entry.

More Jev resources like this

Benchmarking candidate coverage and rejection policy transfer in typed decision models (arXiv 2610.03387)

A benchmark of rejection-policy transfer across Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets, covering artificial omission, natural retrieval misses, and public out-of-scope queries. Source calibration often fails to preserve the target operating point.