CommunityJev Media & Talks
Scoring support calls for policy breaks
Sean Brownell scores sample call transcripts in a harness, including a case where the agent reads out card details, and compares that pass with a frontier-model estimate.
- Format
- Video
- Published by
- Community
- Author
- Sean Brownell
- Last verified
- Oct 2026
Used inJev for Customer Support
Highlights
- One sample transcript is scored in 782 milliseconds. He puts that pass at 3/100 of a cent.
- The failing case is an agent giving out credit card details. He asks Jev to point at the turn.
- He contrasts a 100,000-conversation month at about $52 with a frontier-model estimate near $20,000.
Watch out
The cost and latency figures are his sample runs, not a published benchmark. The transcripts are generated.
Questions about this entry
Short answers drawn from the linked project or page.
No. They come from his own harness on generated transcripts, not a published benchmark, and the entry flags that explicitly. Synthetic transcripts also mean there is no real distribution sitting behind the numbers.
The failing case is an agent reading out credit card details. He asks Jev to point at the turn rather than return a verdict, which is a narrower localization aid than a full moderation decision.
He puts a hundred thousand conversations at roughly $52 against about $20,000 for a frontier model approach, a few hundred times cheaper. That is his extrapolation from sample runs, not a quoted vendor price.
Not on its own. Pointing at a turn localizes a problem, and the policy that acts on it still has to live in your code. The video does not document the threshold he used or how false positives were handled.
Sean Brownell published it. The transcripts are generated samples rather than real calls, and the entry flags the cost and latency figures as his sample runs.
Point at the turn where the agent reads out credit card details, rather than return a verdict. The entry calls that a localization aid, not a full moderation decision.
3/100 of a cent for the 782-millisecond scoring pass, from his own harness on generated transcripts.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
YouTube: Sean Brownell scores sample support calls, including a card-detail leak, and prices a 100,000-call month.
YouTube: Building a contextual brand-safety scorer on Jev from scratch.
X: A website slop detector that checks 35 tells in 243 ms
X: Breaking, golden nugget, or slop
More Jev resources like this
Codevolution's walkthrough of state, questions, and answers, then a customer-message demo in the TypeSafe playground and the TypeScript SDK.
Jev Media & TalksVideo#community#video#tutorial
Community
WitWireAI recreates a spreadsheet that asks Jev to score customer messages by meaning, and labels the clip as a demo rather than a benchmark.
Jev Media & TalksVideo#community#video#classification
Community
AI Test Bench runs four tests on synthetic support tickets in the official TypeSafe playground and states what that small set does not prove.
Jev Media & TalksVideo#community#video#evaluation
Community
Automate with Marc tests Laya, the open decision model people shorthand as Jev for free, on 100 sales leads beside Claude Code.
Jev Media & TalksVideo#community#video#open-models
Community
Turing Post TV tests Jev from early access in the TypeSafe playground, starting from the constraint that it does not write essays or code.
Jev Media & TalksVideo#community#video#evaluation
Community
ML Visualized retells Diogo Almeida's Latent Space point that a launch chart with the new model on top of every row is not something you can trust as a benchmark.
Jev Media & TalksVideo#community#video#evaluation
Community
From the Jev community
Posts from builders shipping with Jev right now.