User repository on GitHub
- Forks
- 0
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Benchmarks & Evaluations
jev_test Guardrail Benchmark
A guardrail benchmark that ports the policy types of Amazon Bedrock Guardrails - six content filters, denied topics, word filters, sensitive information and contextual grounding - to TypeSafe's System One model Jev, then measures latency, cost and classification quality per case.
- Stars
- 0
- Language
- HTML
- Reading time
- 2 min
- Published by
- Community
- Author
- woojin-heo
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open sourceRequires API key
Highlights
- Ports Bedrock Guardrails policy types to Jev: six content filters, denied topics, word filters, PII and contextual grounding.
- Measures latency, cost and classification quality per case, according to the README's opening description.
- Send a state plus typed questions: Noul returns a yes-probability, Choice returns per-option probabilities with confidence, Score a weighted score.
- Its example asks a prompt_attack Noul whether a text tries to make an AI assistant ignore its instructions, using an instructions string and criteria.
- Repo primary language shows as HTML, while the README sample is Python through typesafe_sdk with TYPESAFE_API_KEY read from the environment.
Watch out
README reviewed 2026-10-17; the captured text ends mid-call inside the first Python example, so case counts, harness commands and results tables are not visible. No license or published benchmark numbers appear in the digest; the repo has 0 stars.
Questions about this entry
Short answers drawn from the linked project or page.
A guardrail benchmark repo that ports Amazon Bedrock Guardrails policy types to TypeSafe's System One model Jev and measures latency, cost and classification quality per case.
Six content filters, denied topics, word filters, sensitive information / PII, and contextual grounding.
Through typesafe_sdk's TypeSafeClient, which reads TYPESAFE_API_KEY from the environment, sending a state with a prompt_attack Noul question against an instruction-ignoring text.
Noul gives the probability of a yes/no condition; Choice returns per-option probabilities and confidence; Score returns a weighted score for ordered levels with per-level probabilities.
No results tables were captured; the README excerpt ends mid-call in the first example, so the per-case measurements the harness produces are not visible.
No; the README stresses that Jev does not generate text but returns a probability for each typed question, which is why it is used for guardrail checks.
0 stars, no license listed, primary language HTML per metadata, created and last pushed on 2026-10-04.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Agent Handoff Gate: agents verify what they hand off
X: Jev as an LLM guardrail inside Agentgateway
X: Vercel's fx safety reviewer, measured at 18× faster
Reddit: Early experiments wiring Jev into Pi as a safety layer and model router.
More Jev resources like this
A benchmark comparing TypeSafe Jev with Claude Haiku 4.5 as input guardrails across jailbreak, evasion and benign prompts, measuring detection rates, false positives, latency and cost. At n=1,000 Jev traded some detection for a 328 ms median latency and $0.03 per 1K calls versus Haiku's 2,066 ms and $1.87.
Jev Benchmarks & Evaluations#community#catalog-import#yibie
A benchmarking studio and real-time simulator that pits Jev (remote TypeSafe System One), a local Laya model and the optimal blackjack Basic Strategy against each other on identical deterministic shoes. It scores every deviation turn by turn and runs headless batches of up to 100,000 rounds.
Jev Benchmarks & Evaluations#community#github#blackjack
Community
mrbanana55/decision_blackjackPython
mrbanana55/decision_blackjackPythonAn independent evaluation of TypeSafe's Jev as a prompt-injection detector: 6,991 Jev calls across six tests with three comparison detectors, publishing all code, inputs, per-prompt results and figures, plus a written analysis.
Jev Benchmarks & Evaluations#community#github#security
Harmful/benign classification and answer-choice robustness for five type-safe models: 83,183 source requests from XSTest, OR-Bench and HarmBench in original, reversed and order-only variants.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
An independent Laya-versus-Jev comparison on one RTX 5090: phishing detection over 2,000 emails, the typed-decisions suite, latency scaling to 100 questions per pass and eleven probe assertions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
A Python benchmark that evaluates LLMs' reasoning and instruction-following by making them play chess against a random player or the Komodo Dragon engine in multi-turn dialogs, separating chess skill from protocol durability.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
maxim-saplin/llm_chessPython
maxim-saplin/llm_chessPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.