Moderation is not a single toxicity score. It is a set of decisions about what to do with a piece of content: show it, label it, reduce its reach, hide it, remove it, or send it to a human. A model that answers "is this hateful" gives you one input to that decision. It does not make the decision, and treating it as if it does is how moderation systems produce both false removals and missed harm.
Define the action, not the label
The first design choice is the action set. Jev Content Moderation uses three bands: Approve, Shadow, and Remove. Shadow means the content is hidden but not deleted and the author is not notified. Remove is permanent.
The README's rule is asymmetric on purpose: removing good content is worse than leaving bad content up, so the removal bar sits higher than the approval bar. It cites user rage, bad press, and lost trust as the costs of false removals. The captured README truncates before the question definitions and threshold values, so the implementation is not visible, but the policy shape is the useful part.
That shape maps to a general principle. The more reversible the action, the lower the confidence you need. Shadowing is reversible; deleting is not. A moderation system that only has approve and remove has thrown away its cheapest safety valve.
Accuracy and cost evidence
HATEDECIDE is the most direct evaluation of structured decision models for hate-speech moderation. It tests six decision-model configurations on four hate-speech datasets against specialized, zero-shot, commercial, and supervised baselines.
The headline is nuanced. Commercial LLMs significantly outperform all decision models on only one of the four datasets. On a diagnostic set, the best hosted decision model comes within 1.6 macro-F1 points of the best commercial LLM at approximately 97% lower inference cost.
The negative results are as important as the positive one. Supplying a dataset's definition changed up to 28% of predictions without consistently improving classification. Decomposing the task into more questions significantly improved performance in only 20% of comparisons. The authors identify opportunities for inexpensive moderation while showing that explicit criteria and additional questions do not reliably help.
The caveats: a preprint, a small diagnostic set behind the 1.6-point gap, and an abstract that does not name the hosted model or the individual baselines. Read it as evidence that the cost gap is real, not as a ranking.
Context is the hard part
A word list cannot distinguish "go left, there's a treasure box behind the tree" from useful advice, or a plot twist from a spoiler. The Twitch Chat Filter Cookbook uses three questions per message against a state holding the stream title and game: a five-way category Choice (question, hype, conversation, backseat, spam), a Noul for whether the message is insulting, hateful, or harassing, and a Noul for whether it reveals a plot point or ending.
On 29 hand-written Elden Ring messages, the filter hid 7, blurred 2 spoilers, and highlighted 6 questions, with requests taking a median of about 0.4 seconds. Changing the rules re-ran only the decision function and changed 14 of 29 messages with no new requests. That separation between judgment and policy is the pattern worth copying.
The caveats are the live-channel reality: a toxic message is visible for about 0.4 seconds before the answer arrives, one request at a time cannot keep up with a busy chat, the model was trained mainly on English at publication, and the spoiler check has no record of the streamer's progress.
Dual use and the AUP
The same decision capability that flags harassment can be used to build a censor. Hidden Risks of Jev tests four defensive detection tasks (prompt injection, jailbreak, harmful content, and AI-generated text) alongside misuse scenarios involving jailbreak and model extraction. That is the dual-use frame in one paper.
TypeSafe's Acceptable Use Policy prohibits illegal content, intellectual-property and privacy violations, and phishing or spoofing, and it applies to anyone you allow to use the service. It also bans circumventing safety filters, overriding them, and accessing other customers' data. The list is not exhaustive.
For a moderation team, the AUP is not the constraint; it is the floor. The harder policy question is what your own system is allowed to do with a borderline classification. A model that returns a 0.51 probability of hate should not trigger a permanent ban. It should trigger human review, or a lower-visibility action, or nothing.
Adversarial rephrasing
Same Attack, Different Judge is the best independent evidence on input moderation. It runs 50 attack variants from 13 families under one policy. Eight forgeries of Jev's structured input bypassed none of its moderation decisions, and a sentence asking only why someone might make a harmful request bypassed 9% of its blocks, against 56% for GPT-6 Astra. Jev was the fastest judge tested.
It also flagged the most benign requests, at 27%. That false-positive rate is the moderation-specific cost. A judge that is hard to attack but over-flags will bury reviewers and suppress legitimate speech.
TokenBreak is the other failure mode. HiddenLayer's follow-up reports that both its test model and Jev flagged all 30 toxic prompts before edits; after character insertions that preserved meaning, many slipped past both. One prompt's toxicity score fell by more than two thirds and Jev stopped flagging it. HiddenLayer is a vendor, the comparative model was a converted Qwen model, and the test was black-box. The finding still holds as a class of attack: edit the form, keep the meaning, and watch the score drop.
User-facing filters vs platform decisions
toXic Filter is a user-controlled Chrome extension. It classifies X posts into six categories (discrimination, provocation, hatred, calls to attack, cynicism, resignation) plus an insufficient-context check. A post is hidden when an enabled category passes 70% by default, unless insufficient context is at or above 65%; posts over 6,000 combined characters are never classified. It sends post text near the viewport and quoted text to TypeSafe, runs at most 300 submissions per minute across tabs with 8 in parallel, and bills the user's own account.
That is a good fit for user preference: reversible, local to the user, no platform-wide consequence.
Wolf Defender Jev is the other end: a 151M-parameter Jev-like model for agent security, not content moderation. It reports 92.8 balanced accuracy and 5.4% false positives on 10,576 held-out decisions, at 39 ms for a 51-token scene on an Apple M4 CPU. The caveats matter: it trails Clef-flash on 10 external benchmarks, 85.1 to 90.2, and XSTest over-refusal shows a 35.2% false-positive rate. Small models are attractive for on-device moderation, but their calibration on the population you care about is an empirical question.
Operating a moderation queue
The queue needs an appeal path, a sampling audit, and a policy owner. Log the model version, the question text, the action taken, and the reviewer override. Sample both the escalated and the auto-approved cases; errors hide in both.
jev-guard is not a moderation tool, but its post-tool scan pattern is relevant if a moderator-facing agent reads reported content. Scan retrieved text for instructions before an agent acts on it, and remember hits for the rest of the session. The guardrails article covers the input boundary.
The rejection-design article covers how to make an "insufficient context" outcome useful, and the confidence-gates article covers thresholding. Moderation improves when the system is allowed to choose a smaller action under uncertainty.
