Moderation at volume is a throughput problem and an asymmetry problem at the same time. Every message needs an answer in a few hundred milliseconds, and the cost of hiding good content is not the same as the cost of leaving bad content up. The systems that work make both facts explicit.
That means more than a toxicity score. It means several narrow questions per message, a rule layer that maps answers to visible actions, and thresholds that reflect what each action costs.
One message, three questions
The Twitch chat filter cookbook sends one request per message with three questions against the same state: a five-way category Choice (question, hype, conversation, backseat, spam), a Noul for whether the message is insulting, hateful, or harassing, and a Noul for whether it reveals a plot point. The state carries the stream title, the game, and the message, because "is this a spoiler" depends on what is being played.
On 29 hand-written Elden Ring messages, the filter hid 7, blurred 2 spoilers, and highlighted 6 questions. Requests took a median of about 0.4 seconds, with eight in flight. The examples are deliberately not keyword cases: "go left, there's a treasure box behind the tree" is unrequested gameplay advice, and a plot twist is a spoiler without a single bad word.
The caveat is a live-channel fact rather than a model defect: a message waits about 0.4 seconds for its answer, so a toxic line is visible until then, and one request at a time cannot keep up with a busy chat. The spoiler check also has no record of the streamer's progress. Both are reasons to treat the filter as a layer, not a guarantee.
Put the rule in decide(), not in the prompt
The cookbook's best design choice is that the questions describe the message and a function called decide() turns the answers into what the viewer sees: show, highlight, dim, hide, or blur. At or above the 0.7 toxic threshold the message is hidden; a category whose rule is hide is hidden; at or above the 0.7 spoiler threshold it is blurred; everything else gets its category's rule.
Because the rules live in code, changing them re-runs only decide(). In the example, quieter rules changed 14 of 29 messages with zero new requests. That is the same separation the confidence-gated routing pattern recommends: the answer says what the message is, and a separate policy says what to do about it. A confidence gate belongs next to the action, where it can be reviewed and changed.
Asymmetric bands beat binary remove
jev-content-mod makes the asymmetry explicit with three bands: Approve, Shadow (hidden but reversible, author not notified), and Remove (permanent). Its README states the rule directly: removing good content is worse than leaving bad content up, so the removal bar sits higher than the approval bar.
The repository is a small Python 3.12 project under MIT, with zero stars. The captured README cuts off before the question definitions and threshold values, so the band structure is the transferable idea, not a configuration you can copy. The design principle is still sound: give the system a reversible middle action, and reserve the irreversible one for a higher bar.
What leaves the browser
toXic Filter is a Chrome Manifest V3 extension that classifies X posts into six expression categories plus an insufficient-context check. A post is hidden when an enabled category passes the threshold, 70% by default, unless insufficient context is at or above 65%. It sends seven questions to jev-latest and runs at most 300 submissions per minute across all tabs, eight in parallel, with one retry on 429 or 529. Posts over 6,000 combined characters are never classified, and results stay in memory for at most ten minutes or 1,000 items.
The privacy boundary is the part to read carefully. Post text near the viewport and quoted text are sent to TypeSafe, which can include names and URLs, and posts from visible non-public accounts are in scope. Images, videos, and DMs are not. The token lives in storage.local, not Chrome Sync. The README warns that judgments can be wrong and that the extension does not judge a post's correctness or its poster's character.
Social Monitor is a different kind of tool: a full-stack collector for X, Reddit, Hacker News, RSS, and GitHub that ranks posts and writes cited summaries. Its repository description says the feed is based on Jev, but the captured README never names the model, endpoint, or decision. Treat that integration as stated but undocumented. Several providers are beta, X uses a private collector, and no benchmarks or pricing appear.
What independent evaluations find
HATEDECIDE evaluates six decision-model configurations on four hate-speech datasets against specialized, zero-shot, commercial, and supervised baselines. Commercial LLMs significantly outperformed all decision models on only one of the four datasets. Supplying a dataset's definition changed up to 28% of predictions without consistently improving classification, and question decomposition significantly improved performance in only 20% of comparisons. On a diagnostic set, the best hosted decision model came within 1.6 macro-F1 points of the best commercial LLM at roughly 97% lower inference cost.
That last number is the one vendors quote, so read the caveat with it: the diagnostic set supporting the 1.6-point gap is small relative to the four datasets, and the abstract does not name the hosted model or the baselines individually. The paper is an arXiv preprint from October 2026, not a product evaluation.
Same Attack, Different Judge adds the adversarial view. It sends identical payloads under identical policies across moderation, injection detection, routing, and phishing, with 50 attack variants from 13 families. On input moderation, eight forgeries of Jev's structured input bypassed none of its decisions, and a "why harm" sentence bypassed 9% against 56% for GPT-6 Astra. On phishing, forgeries bypassed 19–29% of the emails Jev flagged, and unattacked Jev detected only 62% against 77–81% for the GLM judges. Jev was the fastest judge but also flagged the most benign requests, at 27%. The authors' conclusion is precise: Jev resists attacks on tasks with a clear rule to check, and is more exposed on tasks that call for an overall judgment.
Small models for the firehose
Wolf Defender Jev is a 151M-parameter fine-tune of mmBERT-small for agent security, not chat moderation, but it shows the edge option. It returns calibrated probabilities for choice, Noul, and Score questions over a serialized agent scene. The card reports 39 ms for a 51-token scene on an Apple M4 CPU and 92.8 balanced accuracy at 5.4% false positives on 10,576 held-out decisions, ahead of Clef-flash's 81.0 and 10.7 on its own panel.
Every one of those numbers is self-reported. On ten external benchmarks, Clef-flash leads 90.2 to 85.1, and XSTest over-refusal shows a 35.2% false-positive rate. Planted approval notes or forged trusted blocks flip at most 0.7% of detected attacks when the scene carries agent context. It is MIT-licensed and fits on a laptop.
Tune the gate, not just the classifier
The confidence page is blunt about the limits. Choice and Score answers carry a confidence statistic, but Noul answers carry only a single probability, and confidence is not permission: a high-confidence answer can still be wrong. Thresholds depend on your domain and the model's performance on your use case, so start conservative and tune against labeled data.
For a firehose, that means separate thresholds for hide, blur, shadow, and escalate; a human review path for the reversible middle; and a fail-open or fail-closed decision made per action rather than globally. The moderation decisions article covers the policy side, and the guardrails article covers adversarial input. The classifier is only one part of the system; the rule layer is what decides whether a wrong answer becomes a visible mistake.
