Phishing and scam screening is not a classification leaderboard. It is a queue with two expensive errors and a fixed review capacity. The useful question is not model accuracy but the precision and recall the system operates at, and what happens to cases it cannot decide.
Two errors and a third action
A screening system can allow, block, or escalate. The third action is what makes the design work, because a binary classifier forces every uncertain case into one of the two errors.
| Action | Error if wrong | Typical cost |
|---|---|---|
| Allow | False negative | Compromised account, credential theft, fraud |
| Block | False positive | Lost legitimate mail, support tickets, missed deadlines |
| Escalate | Review time | Analyst capacity, delayed delivery |
Phishing pushes the threshold toward blocking, but a filter that quarantines too much gets disabled by users. The right operating point depends on the population and the review capacity, not on a generic accuracy figure.
The uncomfortable benchmark
Same Attack, Different Judge is the most useful independent comparison because it sends the same payloads to every system under the same policy. It runs 50 attack variants from 13 families against Jev, five GPT and GLM judges, and two local classifiers across four tasks.
The phishing result is not flattering. Even without attack, Jev detected 62% of phishing emails, against 77-81% for the GLM judges. Forgeries bypassed 19-29% of the phishing emails it did flag. Jev was the fastest judge tested, but it also flagged the most benign requests, at 27%.
That is the number to design around: Jev is a component in a selective-escalation design, not a standalone detector. On input moderation the picture is different: eight forgeries of Jev's structured input bypassed none of its moderation decisions, while a sentence asking why someone might make a harmful request bypassed 9% of its blocks, against 56% for GPT-6 Astra. The authors' conclusion is that Jev resists attacks on tasks with a clear rule to check and is more exposed on tasks that call for an overall judgment. Phishing is the second kind.
Scam calls: calibration and timing
CallScreenBench is a different task with a different result. It evaluates one-pass scam-call screening across 41 held-out scenarios, producing 577 per-turn decisions. A three-seed ensemble of LoRA-tuned Qwen3-4B reaches AUROC .974 with calibration error .052 and no false alarms on legitimate calls, at 64.5 ms per decision on one consumer GPU. That is 4.9x lower than the same backbone fine-tuned to generate its answer, and decisions came 1.14 turns earlier under the same hang-up rule.
The recipe is Jev-like rather than Jev: the model computes a temperature-scaled softmax over two answer-label logits, with no generated text. The authors say the gain is in the readout and calibration, not accuracy, because a fine-tuned ModernBERT encoder is not significantly worse. The caveats are explicit: the recipe was selected with test-set exposure, all callers are synthetic, and the non-inferiority result against MiniMax-M3 uses a pre-registered .02 margin.
The lesson transfers even if the model does not: a per-turn probability plus a calibrated threshold beats a generated verdict, and latency matters during the call.
Email pipelines in the wild
The open email tools split into two camps: prototypes with benchmark numbers and production-shaped scripts with data-handling rules.
The Phishing Spam Jev-Like Classifier is the prototype. It is a DistilBERT model trained on 4,675 SpamAssassin messages, evaluated on an 890-message held-out split: accuracy 0.987, spam F1 0.983, spam PR-AUC 0.998, six spam false negatives. The card is clear that it is a spam classifier, not a phishing detector, that its confidence is an estimate rather than a calibrated probability, and that it does not authenticate senders or check SPF, DKIM, or DMARC.
Jev Mail Filter is the production-shaped script. It runs as a single Google Apps Script file of about 300 lines, sends each new Gmail thread to Jev in one request, and applies plain-English rules as labels, priority stars, Google Tasks deadlines, and phishing flags. It polls every 10 minutes, sends the sender, subject, and the first 2,000 characters of the text, and never sends or deletes anything. If Jev is unsure, nothing happens.
Jev Mail Triage takes the privacy-first route on macOS. It reads a Proton Mail Bridge inbox, asks two questions per message (unsolicited spam, scam or phishing), and records answers in dry-run by default. Mail moves to a Jev Review folder only if you enable it, and nothing is ever deleted. An optional local gate decides obvious cases on the Mac so fewer messages reach TypeSafe.
jev-gmail-ai-spam-filter-and-labeling is the most thoughtful of the three on automation policy. It uses up to twelve plain-English labels with a safe review bucket, checks metadata and a snippet first, and sends full content only when confidence demands it. Archiving requires a label marked archive-eligible and a confidence above your threshold. It never deletes and never moves mail to Spam or Trash, and it starts in preview.
jev-ai-emails-flow shows the same pattern in a corporate-actions pipeline. A Noul ingestion gate decides whether attachments need OCR, routed to Gemini only above a 0.50 probability, and a core Choice classifies the event as cash dividend, M&A, stock dividend, ticker change, or spam. That is a useful separation: the model gates the expensive step, and the classifier handles the event type.
Escalation and consensus
Jev Consensus Gate is a reference design for the escalate path. It asks three independent questions per proposed action: safety, legitimacy, and adversarial framing. Each call is blind to the others. All three must allow; any single block blocks; any escalate routes to a human. Three calls cost roughly $0.00006 per action at $0.042 per million input tokens, about 3x the input cost of a single call.
For email, the analogous design is cheaper: one call for routine decisions, a second framing only for borderline cases, and a human queue for the rest. The asymmetric rule is the part worth copying: a single confident veto wins, a single allow does not.
Data handling is part of the decision
Every one of these tools sends message content to a hosted API. The amounts differ.
| Tool | What leaves the machine | Default action |
|---|---|---|
| Jev Mail Filter | Sender, subject, first 2,000 characters | Labels, stars, tasks; never sends or deletes |
| Jev Mail Triage | Sender, subject, shortened body, link addresses | Records only; moves only if enabled |
| Gmail labeling | Metadata and snippet, then full text when needed | Labels, archive only if eligible |
| Phishing classifier | Runs locally | Binary label, no external call |
If the mailbox is regulated, the retention terms of the API matter as much as the filter's accuracy. The data-retention article covers the legal set and the per-surface verification. The PII checker is a useful cautionary example: it sends scanned text to the API, and its own README says to use it only on data you are allowed to hand to a third party.
Measure your queue
Track false positives per 1,000 legitimate messages, detection at your actual review budget, median time to decision, and appeal rate. Run a reworded-attack set, not just known samples, in the languages your users receive.
jev-guard is not a mail tool, but its audit mode is the right rollout pattern: log what would have been blocked without blocking anything, then enable enforcement. Jev Mail Triage does the same with dry-run.
The confidence-gates article covers thresholding in more depth, and the rejection-design article covers what a useful "needs review" state looks like. Screening works when the system is allowed to say it does not know.
