Testing Jev on public and private data: classifier or filter?

On many-way pick-one and on long input. Banking77 with 77 intent options in a single question is the one public set where both LLMs beat it: 76.0 against 78.7 for gpt-5.4-mini and 81.7 for gpt-5.6-luna. It won the crisp short-text sets (Enron spam 98.7, SST-2 95.7, AG News 91.3), but a whole-document read scored 68.6% and no prompt fixed it.

Real at moderate concurrency, with a tail. At 20 calls in flight the median was 0.8 to 0.9s and the 95th percentile under 1.1s, against 1.4 to 5.0s medians for the two OpenAI models, and Jev's time barely moved as option count grew. At 100 calls in flight on the author's own pages, a few per thousand took 10 to 35 seconds.

Both, and the split matters. The four public sets are scored against real dataset labels with per-item answers and scoring code published, so they are checkable. The production pipeline numbers are scored against what actually happened afterwards, but those decisions came from a private pipeline nobody can inspect, so treat them as directional evidence only.

Put it in front. The verdict is filter, not replacement: on production page gates it agreed with outcomes 96 to 98% of the time and let 25 to 60% of pages past the classifier with nothing lost. The threshold is empirical rather than a default: set the drop line at half the lowest P(yes) any known positive received, fit it on one part of the data and check on another fold.

About two days and 16,000 calls: 300 random items from each of Enron spam, SST-2, AG News, and Banking77, then a few thousand real decisions from production pipelines, scored against what actually happened rather than another model's opinion.

On the public sets Jev was 0.8-0.9 s median (95th percentile 0.9-1.1 s) versus 1.4-5.0 s for gpt-5.4-mini and gpt-5.6-luna, measured with 20 calls in flight. The LLMs get slower as options grow because they write a probability per option.

On tasks it can do, confident answers were right 90-100% of the time and the middle is a coin flip. It is best on short input with crisp labels, and accuracy and confidence fall together as inputs get longer, labels fuzzier, or the question carries more business logic.