Article
CommunityJev Benchmarks & Evaluations
Humanized but Still True? Auditing Facts and Detection in AI-Text Humanizers
arXiv paper auditing AI-text humanizers for fact retention and detector evasion, using a rule-based checker and a zero-shot Jev question to separate human originals from humanized rewrites.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Rule-based checker validated by two AI annotators: precision 0.84-1.00, recall 0.93-0.95; no human annotation.
- All 11 commercial humanizers changed or dropped at least one number, date, name, or quoted span in 35-65% of documents containing one.
- Jev, asked one zero-shot question, separated human originals from humanized rewrites with AUROC 0.864 (0.827-0.902).
- Jev scored 0.18-0.22 AUROC higher than Binoculars, Fast-DetectGPT, and RADAR, but called 24% of human originals AI at a 0.5 threshold.
- Checker and per-item results are released; hypotheses, data rules, and analyses were frozen before main runs.
Questions about this entry
Short answers drawn from the linked project or page.
It checks whether numbers, dates, person and organization names, and quoted spans survive AI humanizer rewrites, and it evaluates detector evasion on federal texts with matched human, AI-styled, and humanized versions.
Jev, a general-purpose decision model, was asked a single zero-shot question to separate human originals from humanized rewrites, achieving AUROC 0.864 (0.827-0.902).
It checked retention of numbers, dates, person and organization names, and quoted spans. Two AI annotators validated it with precision 0.84-1.00 and recall 0.93-0.95; no human annotation was used.
In the October 2026 leaderboard cycle, all 11 commercial tools changed or dropped at least one checked unit in 35-65% of documents that contained one. Counting numbers and names alone, 10 of 11 altered at least one in 20% or more of documents.
Jev scored 0.18-0.22 higher AUROC than those detectors. On AI-styled rewrites by 2026 models, both perplexity-based detectors scored below chance (AUROC 0.37 and 0.39), also with line breaks removed.
At a 0.5 threshold, Jev called 24% of human originals AI. Unit retention and detector evasion were not clearly related across tools (Spearman rho = -0.46, 95% CI -0.65 to 0.08).
The checker and per-item results are released. Hypotheses, data rules, and analyses were frozen before the main runs, and two earlier data contacts are disclosed.
More Jev resources like this
Same Facts, Different Remedy: Linguistic Profiling and Arbitrariness in LLM and System One Decisions
arXiv paper auditing three remedy-decision systems on US consumer financial complaints: GPT-4o-mini, GPT-5-mini, and Jev, a probabilistic System One model returning a distribution over remedies. It separates group-scale disparities from individual decision instability.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Empirical study evaluating four specialized decision models on zero-shot text classification in Vietnamese across 13 classification settings and six task groups, testing English, Vietnamese, and bilingual label descriptions plus teencode, typo, and diacritic perturbations.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#classification
Community
arXiv paper presenting AbductionBench, a unified framework and taxonomy for abductive reasoning tasks, instantiated on 27 datasets across ten domains and used to evaluate eight LLMs from four model families on single-choice explanation selection.
Jev Benchmarks & EvaluationsArticle#arxiv#benchmark#evaluation
Community
Open, deterministic audit benchmark for System 1 decision models, covering calibration under label-preserving perturbations, cross-question interference, and post-hoc attribution faithfulness across five open-weight models and generative baselines.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper evaluating 67 systems, including 47 decision models and 20 LLMs, on 43 Arabic tasks across 9 task families under a shared zero-shot protocol. Jev 1.14.0 scores 69.24 against 68.77 for Jev 1.13.0 and 71.81 for Gemma-4-31B.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Multilingual benchmark of nine suites in 122 languages that evaluates typed decision models such as Jev, Clef, and pplx-decider, and tests whether confidence thresholds fitted on English transfer to other languages.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.