What does this benchmark compare?

TypeSafe Jev against Claude Haiku 4.5 as input safety classifiers for enterprise AI: both evaluate prompts for injection attempts, jailbreaks and harmful intent, and the benchmark measures detection, false positives, latency and cost.

What datasets does it use?

Jailbreak-PromptBank for attacks, Mindgard's evaded prompt-injection and jailbreak samples for evasion, and PHTest for benign pseudo-harmful prompts, sampled at roughly 53% attack, 13% evasion and 33% benign with pinned dataset revisions.

How does the benchmark query Jev?

Through TypeSafe's System One API with typed questions: three Noul questions for injection, jailbreak and harmful intent plus one Choice risk level, flagging unsafe when any Noul exceeds 0.5 or the risk level is dangerous.

What were the headline results?

At n=1,000: Jev detected 91.9% versus Haiku's 96.6%, with 78.2% versus 94.0% on evaded prompts, lower false positives (9.6% vs 14.1%), a 328 ms versus 2,066 ms median latency, and $0.03 versus $1.87 per 1K calls.

Where do the cost figures come from?

The README computes them from Jev input pricing of $0.042 per 1M tokens and Anthropic Haiku 4.5 pricing of $1.00 per 1M input and $5.00 per 1M output, as of October 2026. Your actual bill depends on prompt lengths.

How do I run the benchmark myself?

Clone the repo, run uv sync, copy .env.example to .env and fill in TYPESAFE_API_KEY, ANTHROPIC_API_KEY and HF_TOKEN, then run uv run python benchmark.py. Use -n to scale the case count or --all for every available prompt.

What does each run produce, and what are the limits?

A timestamped results directory with summary.json, detail.csv and raw_results.jsonl checkpointed every 50 calls. It is a single-author benchmark on public datasets; the README's tables are the only published numbers and it reports a 1,000-case default run.

Jev directory

CommunityJev Benchmarks & Evaluations

Jev as a Guardrail

A benchmark comparing TypeSafe Jev with Claude Haiku 4.5 as input guardrails across jailbreak, evasion and benign prompts, measuring detection rates, false positives, latency and cost. At n=1,000 Jev traded some detection for a 328 ms median latency and $0.03 per 1K calls versus Haiku's 2,066 ms and $1.87.

See highlights Open github.com

Stars: 0
Language: Python
License: MIT
Reading time: 2 min
Published by: Community
Author: rahulbansalc6414
Last verified: Oct 2026

rahulbansalc6414/jev-as-a-guardrail

User repository on GitHub

View on GitHub

Benchmark: TypeSafe Jev vs Claude Haiku 4.5 as input guardrails. Compares detection rates, false positives, latency, and cost across 1500 jailbreak, evasion, and benign prompts.

Forks: 0
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.

Tagscommunity catalog-import yibie benchmark guardrails

Works forJev for Evaluation Jev for Moderation

Used inJev for Security & Privacy

Languages: Python
Platforms: CLI
Capabilities: Open sourceRequires API key

Highlights

Datasets: Jailbreak-PromptBank attacks, Mindgard evaded injections and PHTest pseudo-harmful benign prompts, sampled 53/13/33 with pinned revisions.
Jev asks three Noul questions plus one Choice risk level and flags unsafe when any Noul exceeds 0.5 or the level is dangerous.
Results at n=1,000: detection 91.9% vs 96.6%, evasion 78.2% vs 94.0%, false positives 9.6% vs 14.1%, median latency 328ms vs 2,066ms.
Cost per 1K calls: $0.03 for Jev at $0.042/1M input tokens versus $1.87 for Haiku 4.5 at $1.00/1M input and $5.00/1M output.
Requires TYPESAFE_API_KEY, ANTHROPIC_API_KEY and HF_TOKEN; runs are checkpointed every 50 calls into timestamped results directories.

Quickstart

bash

uv sync
uv run python benchmark.py -n 100

Watch out

Grounded in the complete repository README and GitHub metadata reviewed on 2026-10-15; results are the README's n=1,000 run with pricing as of October 2026, the digest description mentions 1,500 prompts while the published run used 1,000, and all three API keys are required. MIT, 0 stars.

Questions about this entry

Short answers drawn from the linked project or page.

Reactions & coverage

Posts, threads, and videos about this entry from around the web.

X: Agent Handoff Gate: agents verify what they hand off

X: Jev as an LLM guardrail inside Agentgateway

X: Vercel's fx safety reviewer, measured at 18× faster

Reddit: Early experiments wiring Jev into Pi as a safety layer and model router.

From the Jev guides

Original write-ups on what this entry is for.

The trust ledger: measuring confidence your domain has earned

Oct 16, 2026

ASSAY-001 calibrated Jev on CLINC150 but found it overconfident on Banking77 — one audit, two verdicts. How calibration research measures the model-and-domain pair, discounts stated confidence, and signs the evidence.

Jev Benchmarks & Evaluations#calibration#evaluation#open-models

Read article

From the Jev AI model community

Posts from builders shipping with Jev right now.

Follow @typesafeai