Syntheme/beyond-answer-confidence
Organization repository on GitHub
- Forks
- 0
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.
CommunityJev Benchmarks & Evaluations
Beyond Answer Confidence: A Controlled Audit of Self-Knowledge in a Black-Box Decision Model (arXiv 2610.01006)
An audit of whether Jev's confidence reflects missing knowledge rather than chance, using over 15 public datasets and 6 generated task families with paired interventions. Jev's confidence is calibrated on familiar closed-choice tasks but fails as an indicator of missing knowledge.
- Stars
- 4
- Language
- Python
- License
- APACHE-2.0
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
- Languages
- Python
Highlights
- Uses over 15 public datasets and 6 generated task families with paired interventions.
- With no answer-relevant information, Jev assigns up to 0.80 to a salient option.
- On news beyond an observed knowledge boundary, confidence exceeds accuracy by 0.21–0.33, a gap recalibration does not close.
- Whether the evidence suffices is read at AUROC 0.95, against 0.85 for confidence on the same items.
- Asking whether Jev knows the answer flags fabricated entities at 0.91 but shows no advantage with realistic names or removed dates.
Watch out
arXiv preprint posted October 1, 2026 (cs.AI, cs.LG) by Sharath M Shankaranarayana, Davor Runje, and Jan Jannink. The audit targets one black-box decision model; the authors state that black-box knowledge audits need explicit controls for surface cues, since apparent self-knowledge can depend on them.
Questions about this entry
Short answers drawn from the linked project or page.
Whether low confidence reflects chance or missing knowledge, and whether confidence falls when the model moves beyond what it knows; calibration alone establishes neither.
It assigns up to 0.80 to a salient option even with no answer-relevant information available for the fixed item.
On news beyond an observed knowledge boundary, confidence exceeds accuracy by 0.21–0.33, a gap that recalibration on earlier months does not close.
Targeted yes/no questions: whether an outcome is settled reads at AUROC 1.00 and whether the evidence suffices at 0.95, against 0.85 for confidence on the same items.
It appears to flag fabricated entities and post-boundary news at 0.91, but with realistic names or with dates removed it shows no advantage over answer uncertainty.
They need explicit controls for surface cues, since the audit shows apparent self-knowledge results can depend on those cues.
The code is published at github.com/Syntheme/beyond-answer-confidence.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
Introduces Sys1Cal-v1, a dataset of True/False questions about propositions whose exact probability is known by construction, queried through Jev's three primitives and scored by total variation from ground truth. Recovering a suppressed third value raises median soft accuracy from 0.771 to 0.978.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A benchmark for typed decision models such as Jev, built from seven policy-labelled generators and nine evaluation suites. It reports accuracy as medians across paraphrases, calibration against a matched noise floor, and realized cost under asymmetric cost matrices.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
0GitHub stars
A benchmark of rejection-policy transfer across Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets, covering artificial omission, natural retrieval misses, and public out-of-scope queries. Source calibration often fails to preserve the target operating point.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
A study posing speech-neuroprosthesis rescoring as a single typed Jev decision. On 978 held-out sentences from a participant with ALS, Jev reaches 7.5% word error versus 7.8% for OPT-6.7b and Qwen2.5-7B, at USD 0.07 per thousand sentences with no GPU needed.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
An evaluation of JEV-as-a-Judge, a decision-only judge that returns label probabilities and uses its confidence to accept a verdict or escalate to a reasoning judge. Against sixteen generative and reward-model judges it comes within three points of GPT-6 where a verdict can be read off the text, at 0.36% of its fee.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
An independent reproduction of the shipped Laya Typed-Decisions checkpoint, a 421M-parameter ModernBERT-large assessor, finding it uniformly under-confident rather than over-confident. A single fitted temperature removes most miscalibration, while the preregistered isotonic selection failed its held-out contrast.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.