Question 1

So does Jev actually detect AI-written pages, or not?

Accepted Answer

Not in any useful sense, and that is the finding. Across the full run of 4,579 pages Jev's Noul reached 0.741 accuracy with 0.257 precision and 0.082 recall, below the 77.6% always-human baseline. Sweeping the threshold rescues recall only by sliding toward calling everything an agent, and precision stays pinned at the 22.4% base rate the whole way, because the agent and human score distributions sit almost on top of each other with medians of 0.44 and 0.42.

Question 2

Why would a classifier work at all if a regex finds nothing in the prose?

Accepted Answer

Because the target is provenance, not style. Labels come from per-page author names - a non-bracketed A... pseudonym for agents, a bracketed human name otherwise. A regex hunting I am an AI, language model, as an AI assistant, GPT or Claude scores precision 0.000 and recall 0.000, meaning it predicts human for every page and matches the majority baseline's 0.776 accuracy for exactly that reason. The README's position is that the page text does not carry the signal at all.

Question 3

How does the local LLM comparison hold up - is that recall number real skill?

Accepted Answer

The README calls it degenerate rather than skill. The local LLM calls 4,394 of 4,579 pages agent, so its 0.974 recall comes with 0.227 precision, which sits below the 22.4% base rate, and 0.252 accuracy, the worst row in the table. It is also about four times slower, averaging 3620 ms per page against Jev's 830 ms. Two timeouts were excluded, so its n is 4577 against Jev's 4579.

Question 4

Can I rerun this without taking on someone's wiki dump?

Accepted Answer

Yes, and that is the intended path. The corpus has no stated license on its page text, so the raw jsonl.gz dumps are not committed; fetch_data.sh downloads them at runtime, about 10 MB gzipped, and you point COLLUSION_LABELS and COLLUSION_REVISIONS at the files. Only the label join and the code are open. You will need a TypeSafe key and a local Qwen on port 8001, and the README suggests compare.py --limit 200 as a smoke test before the full run.

Question 5

What is the dataset, and what is the majority baseline?

Accepted Answer

The collusion.wiki corpus documents a July 2026 episode in which autonomous AI agents edited a wiki, with per-page author labels: 1,026 agent pages and 3,553 human pages. Always answering human scores 77.6% accuracy, which is the bar any real classifier has to beat.

Question 6

How do I run the benchmark myself?

Accepted Answer

Fetch the corpus with ./fetch_data.sh, point COLLUSION_LABELS and COLLUSION_REVISIONS at the dumps, sanity-check the join with python3 ground_truth.py and see the content floor with baseline_regex.py. The full comparison needs a TYPESAFE_API_KEY and a local Qwen served on port 8001; start with compare.py --limit 200.

Question 7

Why aren't the corpus dumps included in the repository?

Accepted Answer

The page text has no stated license, so raw dumps are not committed; fetch_data.sh pulls them at runtime and only the join logic and code are open. Jev latency in the run was network-bound at roughly 700 ms per call.

Jev Trace Classifier

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev guides

From the Jev community