A Live BS Meter on a Debate

Model judgment, not truth, and the author says so in the post: the same five questions go to both candidates, clips are picked by one fixed rule, and it is explicitly not a fact-check. Replies push the same point, noting that opinions end up rated alongside factual claims.

The full run was 1,191 Jev calls, 1.18M tokens, and $0.0497 at a 415 ms median. With five yes or no questions per sentence for each speaker, that is five calls per sentence per candidate, which is why the token count is large relative to the price.

By a single fixed rule rather than a model, which makes the run reproducible but crude. The post does not describe what makes a sentence a candidate, so a rhetorical one-liner and a substantive claim end up treated the same way before the questions are even asked.

The author rules it out by calling it not a fact-check. A yes or no probability on whether a statement is false is a model judgment with no grounding step, so it can be confidently wrong about true statements, which is the same objection raised in the replies.

Chetaslua describes himself as an AI researcher and reporter who tests new models on launch day, builds working demos, and posts prompts and results, with a code link. That context matters because the BS meter is a launch-week demo rather than a study.

One reply says it is giving BS ratings on opinions rather than facts and calls it a fail, and another argues this is probably not a System 1 task and asks how Jev can judge these things at all. A third jokes about one candidate's meter not being permanently maxed out.

Posted 17 September 2026 at 6:35 AM, the post shows 192.5K views with 77 replies, 65 reposts, 1.5K likes and 753 bookmarks, which is the context for how the 1,191-call run was received.