Which alignment failures does RLCDAlignBench cover?

Ten: sycophancy, jailbreaks, deception, prompt injection, hallucination, privacy violation, social bias, reward hacking, concealing uncertainty, and power seeking.

What data does the benchmark span?

It spans 44 benchmarks and five target models, labelled by each benchmark's scorer and, on two of the benchmarks, by human annotators.

What AUROC does a single generic question reach?

One generic question reaches a median AUROC of 0.886 zero-shot and beats supervised baselines on most benchmarks, according to the paper.

Do question wording and context both matter?

Question wording mattered little, while context mattered more, mostly through input fields that encode the label, which is why the authors vary what Jev is asked and what it sees separately.

How does Jev compare with scorers on human agreement?

Jev matches the reference scorer's agreement with human labels and surfaces label defects in existing benchmarks, the paper reports.

How much cheaper is Jev than LLM judges?

The paper reports Jev costs 63 times less than LLM-judge scorers for the alignment-failure detection setup it benchmarks.

Is RLCDAlignBench's code and data available?

Yes. The abstract lists code and data at github.com/sumleo/RLCDAlignBench for reproducing the zero-shot Jev evaluation.

CommunityJev Benchmarks & Evaluations

Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures (arXiv 2609.29429)

RLCDAlignBench measures Jev as a zero-shot detector of ten alignment failures across 44 benchmarks and five target models. A single generic question reaches median AUROC 0.886 and beats supervised baselines on most benchmarks at 63 times lower cost than LLM judges.

See highlights

Open arxiv.org sumleo/RLCDAlignBench

Stars: 28
Language: Python
License: MIT
Format: Article
Published by: Community
Last verified: Oct 2026

sumleo/RLCDAlignBench

User repository on GitHub

View on GitHub

RLCDAlignBench: 44 alignment-failure detection benchmarks and code for 'Just Ask Jev' (RLCD zero-shot detector of AI alignment failures)

Forks: 4
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.Homepage

Tagscommunity research arxiv alignment calibration evaluation

Works forJev for Evaluation Jev for Moderation

Used inJev for Software & DevTools Jev for Security & Privacy

Highlights

Covers ten alignment failures, from sycophancy and jailbreaks to reward hacking and power seeking, across 44 benchmarks.
Varies question wording and answer type separately from the input fields Jev sees, because many failures are relational.
A single generic question reaches a median AUROC of 0.886 zero-shot and beats supervised baselines on most benchmarks.
Question wording mattered little, while context mattered more, mostly through fields that encode the label.
Jev matches the reference scorer's agreement with human labels and costs 63 times less than LLM-judge scorers.

Watch out

arXiv preprint posted September 24, 2026 (cs.AI, cs.CL, cs.CR). The authors design the benchmark and evaluate Jev on it; ground truth comes mostly from benchmark scorers with humans used on two, and the paper surfaces label defects, so AUROC depends on existing labels and may not transfer to other failures.

Questions about this entry

Short answers drawn from the linked project or page.

Related terms

Glossary definitions related to this entry.

More Jev resources like this

Same Attack, Different Judge: A First Controlled Adversarial Comparison of Jev, LLM Judges, and Purpose-Built Safety Classifiers (arXiv 2610.17820)

A controlled adversarial comparison of Jev against five GPT and GLM judges and two local classifiers under identical payloads and policies. Jev is the most robust on input moderation and the fastest overall, but detects only 62% of phishing emails and flags the most benign requests.