Jev directory
CommunityJev Benchmarks & Evaluations
EvalSafe Security Incidents
Hugging Face dataset from TypeSafe AI pairing security incident telemetry with model decision outputs. Tabular and text parquet rows, English, 1K–10K records, loadable with Datasets, pandas, or Polars.
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Rows pair incident context with OpenAI and Anthropic decision outputs.
- Records span Active Directory, CrowdStrike, Intune, Tanium, Okta, ServiceNow, and Lenel.
- Metadata carries ATT&CK technique IDs such as T1003.002 and an asset tier.
- Distributed as parquet with Datasets, pandas, and Polars listed as libraries.
- Size is listed as 1K–10K rows and the language as English.
Questions about this entry
Short answers drawn from the linked project or page.
It is published on Hugging Face as parquet files with the Datasets, pandas, and Polars libraries listed, so it can be loaded through the Hugging Face datasets library or read directly as parquet.
Rows combine an incident context with telemetry records and model outputs. The example shown includes asset details, records from Active Directory, CrowdStrike, Intune, Tanium, Okta, ServiceNow, and Lenel, plus metadata and provider decision JSON.
The sample rows include outputs labeled openai with model gpt-6-astra and anthropic with model claude-opus-5, both at high reasoning effort and question_mode one, alongside a consensus field.
Metadata fields include an activity identifier such as art/T1003.002-5, a techniques list containing T1003.002, a domain of identity, an attack type of account_takeover, and a fact value of change_absent.
The dataset card lists a size of 1K to 10K rows, English as the language, and Tabular and Text modalities in parquet format.
Decision entries carry a policy_id such as playbook and named actions with JSON arguments; the sample shows a revoke_sessions action selected as the primary action.
The card describes evaluation workflows data containing model decisions and a consensus field. No aggregate accuracy or calibration scores appear in the dataset card shown.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
YouTube: Kubernetes admission control with a decision model, fail-closed.
Hacker News: SafeDep Gryph plus Jev checks for compromised coding-agent behavior.
2 pointsDetecting Compromised AI Coding Agents with Jev and GryphRead the thread on Hacker NewsX: Jev and Laya on security queries
X: Testing Jev for prompt injection
More Jev resources like this
Benchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
Controlled benchmark of Jev against Claude Sonnet 5 on one coding-agent decision: given a bug report, which file needs editing? It scores 200 real SWE-bench Verified issues with cost, latency, and confidence-band routing tables.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
CommunityK624/jev-localization-bench
Spring Boot and Spring AI benchmark comparing Java keyword rules, Jev 1.13.0, and Claude Haiku 5.5 on warehouse goods-receipt triage, with a 200-remark synthetic dataset and a Python analysis script.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
Communitymohammedahmadi1990/jev-triage
A TypeSafe AI Hugging Face dataset of agent trace observability records for evaluation workflows, with cases (111 rows), questions (1.12k rows), and run_results (999 rows) in parquet, including per-provider and consensus decision blocks.
Jev Benchmarks & Evaluations#evaluation#agents#benchmark
Community
Hugging Face dataset from TypeSafe AI of customer-service dialogs with per-policy decisions from OpenAI and Anthropic models and a mean-probability consensus field. Parquet, English, 1K–10K rows.
Jev Benchmarks & Evaluations#evaluation#benchmark#routing
Community
arXiv study applying Jev and Clef-Flash to manufacturing defect detection with as few as two reference images per class, including agentic refinement of text descriptions and probability calibration. Ships replication code, an R decisions package, and a no-code web app.
Jev Benchmarks & EvaluationsArticle#arxiv#calibration#classification
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.