Adversarial robustness is harder to measure for a decision model than a generative one, for structural reasons. Identical requests can return different answers, most available labels come from the model itself, and the API preprocesses each request out of view. A benchmark built for text generation does not transfer. JevAdvBench is the first purpose-built attempt, and its design choices are as instructive as its results.
Why generative benchmarks do not transfer
A generative benchmark scores whether the output contains a target string or satisfies a judge. A typed decision has no text to match; it returns a probability per option, and repeatability is not guaranteed across calls.
The JevAdvBench authors name three problems: request-level nondeterminism, labels that come mostly from the model, and preprocessing that happens before the model sees the request. Their answer is label-free: score each attacked decision against the model's clean answer, and compare that change with an identical re-run. It controls for ordinary noise without pretending the clean answer is ground truth.
What JevAdvBench measures
The benchmark contains 812 typed questions across 66 scenarios and a black-box suite of 9,744 single-edit variants, each editing one part of a request. The authors use billed input tokens to verify that an edit actually reached the model through preprocessing.
The results on jev-1.13.0 are specific:
- Rewording stayed within 1.2 percentage points of the re-run baseline.
- Fields outside the schema never reached the model.
- One unverified opinion appended to the state flipped 12.1% of decisions, statistically tied with the strongest injected command at 10.1%.
- That edit pushed 38% of confident answers below the 0.8 confidence threshold that routes them to human review.
The last number matters. Confidence is not a manipulation detector: an attacker who appends an opinion can move a decision and lower certainty at once, the combination that defeats a naive gate.
The caveats are real: a 33-page preprint, one model version, an author-designed methodology, and thresholds specific to the configurations tested.
Adaptive attacks and leaked judge scores
A static attack suite measures robustness against a fixed list. An adaptive attacker measures something else: how much information the system leaks through its scores.
Jev-Guided Strategy Bandit uses multi-dimensional probabilistic judge feedback to guide prompt optimization. Under a budget of only 10 target-model queries, it averages an 89.8% HarmBench attack success rate across ten open-source settings. It combines continuous jailbreak scoring with targeted refinement, strategy switching, and best-state reseeding, and it was evaluated on two harmful-request datasets across five open-source and three closed-source LLMs.
The paper is an arXiv preprint under review. Its title and defense analysis reference Jev, but the abstract describes only multi-dimensional probabilistic feedback, without naming the implementation. Read it as Jev-style until the setup is detailed; attack success is benchmark-specific and varies with query budget and attacker model.
The design lesson does not depend on those caveats. If an attacker can see a judge's probability, they can optimize against it. Do not expose raw judge scores to untrusted callers, rate-limit repeated probes, and treat a rising success rate as an incident signal.
Comparative black-box results
Same Attack, Different Judge is the most controlled comparison available. It sends the same 50 attack variants from 13 families to Jev, five GPT and GLM judges, and two local classifiers, under one policy, across moderation, injection detection, routing, and phishing.
The pattern is task-dependent:
| Task | Result |
|---|---|
| Input moderation | Eight forgeries of Jev's structured input bypassed none of its decisions; a why-harm sentence bypassed 9%, against 56% for GPT-6 Astra |
| Phishing | Jev detected 62% unattacked, against 77-81% for the GLM judges; forgeries bypassed 19-29% of flagged emails |
| False positives | Jev flagged the most benign requests, at 27% |
| Speed | Jev was the fastest judge tested |
The authors conclude that Jev resists attacks on tasks with a clear rule to check and is more exposed on tasks that call for an overall judgment. That is a more useful claim than "robust" or "vulnerable." It tells you where to put the model and where to put a different control.
The local classifiers failed differently: a statement drawn from training examples made Llama Guard 3 miss 62% of the phishing emails it had previously flagged. A single benchmark number would have hidden that.
Vendor black-box tests
HiddenLayer's research is the vendor test that traveled furthest. On a simulated checkout page, Jev's API gave an attacker-controlled ad's button 91% and the real "Place order" button 9%. HiddenLayer's converted Qwen model went from 99.7% on the real button before the ad to 79.9% on the attacker's button and 19.3% on the real one after. In the toxicity test, both models flagged all 30 toxic prompts before TokenBreak-style edits; after the edits, many slipped past, and one prompt's score fell by more than two thirds.
HiddenLayer sells security products, Jev is a closed API tested from outside, and the comparative model is a conversion rather than Jev's internals. The results are a demonstration of the attack class, not a measurement of the hosted model. Nudge the Judge supplies the application-level version: 81.5% of 228 popular Jev applications showed decision flips across 2,273 test cases, with potential security impact in 98.6% of those flips, and defenses helped little. That too is self-reported and selected.
Contract and regression testing
systemone-probe is a conformance harness for endpoints that speak the POST /v1/systemone protocol. It reports on contract, correctness, calibration, determinism, batching, robustness, and limits. The sample report shows all seven passing. Much of the contract can be checked without labels: probability sums, arg-max agreement, score equals the weighted sum, and confidence reproduced from the answer's probabilities.
The caveats are practical: a 0-star repository, a README that stops before installation, and no published metric definitions. It is a starting point, not a finished product.
jev-consensus-gate adds a runtime control rather than a test. It asks the same action three independent ways (safety, legitimacy, adversarial framing), requires all three to allow, lets any single block veto, and routes any escalate to a human. Three calls cost about $0.00006 per action, roughly 3x the input cost of one call. Consensus is not robustness against a determined attacker, but it catches framing blind spots that a single question misses.
A robustness program
- Pin the model version and the option order. A floating alias makes every result incomparable.
- Treat the state as adversarial content. Run an edit suite that appends opinions, injects commands, rewrites format, and changes option order.
- Measure confidence shifts, not just label flips. The 38% drop below the review threshold is the early-warning signal.
- Keep raw probabilities away from untrusted callers, and rate-limit repeated probes.
- Use a deterministic control for irreversible actions. Consensus or a policy gate, not a higher threshold.
- Re-run the suite on every version change. The reproducibility article covers the pinning mechanics.
- Map reachable state surfaces in the application. JEV Attack Surface Analysis does this for Python, JavaScript, and TypeScript at about $0.01 per repo, ranking candidates rather than proving them.
- Read the jaggedness notes before writing the suite. Counting, dates, negation, and contradictory criteria are separate failure modes from adversarial edits.
The measuring-Jev article covers evaluation design, and the hidden-risks study covers the inference and poisoning side that a black-box attack suite will not find. Robustness is not a property you certify once. It is a test suite you keep running.
