Organization repository on GitHub
Can AI agents resolve production incidents?
- Forks
- 117
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.Homepage
CommunityJev Benchmarks & Evaluations
SREGym
SREGym stands up live Kubernetes clusters so SRE agents can be evaluated against 90 fault scenarios, including OS-level faults and metastable and concurrent failures, in containers that cannot see the grading logic.
- Stars
- 338
- Language
- Python
- License
- MIT
- Published by
- Community
- Author
- SREGym
- Last verified
- Oct 2026
About this project
SREGym: A Benchmarking Platform for SRE Agents. 🔍 Overview SREGym is an AI-native platform to enable the design, development, and evaluation of AI agents for Site Reliability Engineering (SRE). The core idea is to create live system environments for SRE agents to solve real-world SRE problems. SREGym provides a comprehensive SRE benchmark suite with a wide variety of problems for evaluating SRE agents and also for training next-generation AI agents.
Quoted from the project's README. Read the full README.
- Languages
- Python
Highlights
- 90 problems covering everything from AIOpsLab and ITBench plus OS-level faults, metastable failures, and concurrent failures.
- SREGym-Lite is 21 representative problems that run on a Kind cluster with 8 vCPU and 16 GB of memory.
- Agents run in isolated containers built on first run, with network access filtered to the model provider by default.
- Jev review of diagnostic tests stays off unless you set TYPESAFE_API_KEY and pass --jev-model jev-latest.
Watch out
Setup expects SSH and root access to Linux hosts for OS-level cluster configuration, so a managed Kubernetes service does not work out of the box, and a local kind cluster does not support every problem. Container hardening blocks apt-get unless you disable it, and Jev review is optional and sends evidence to TypeSafe. Copy reviewed against the repository README on 2026-10-04, not from running it.
Questions about this entry
Short answers drawn from the linked project or page.
Not out of the box, because setup needs SSH and root on the nodes for OS-level cluster configuration. The README suggests plain VMs or VPS instances added to inventory.yml with the provided Ansible playbook, or a local kind cluster, which does not support every problem.
Start with SREGym-Lite, a curated set of 21 problems with varied difficulty that runs on a Kind setup with 8 vCPU and 16 GB of memory. The full suite is 90 problems, and the --profile svelte option drops components nothing reads.
Not by default: agents always run in isolated Docker containers with no access to SREGym internals such as problem definitions and grading logic. The image is built automatically on first run, and --force-build rebuilds it after you change dependencies or agent code.
No by default. Hardening drops every Linux capability except DAC_OVERRIDE, which container root needs for the host-owned bind mounts, and sets no-new-privileges, which blocks apt-get because it cannot drop to the _apt user. Pass --container-hardening off if the agent needs to install tooling.
The benchmark suites contain 90 SRE problems, supporting all AIOpsLab and ITBench problems plus new ones such as OS-level faults, metastable failures and concurrent failures. SREGym-Lite is a curated set of 17 representative problems that runs on a Kind setup with 8 vCPU and 16 GB of memory and is the recommended starting point.
Requirements are Python 3.12 or newer, Docker, Helm 4.0 or newer, kubectl, uv, and kind for local runs. Clone with --recurse-submodules, run uv sync and uv run prek install. Then set a model provider key and run uv run main.py --agent stratus --model gpt-5, or add --suite sregym-lite.
Jev is disabled by default. To enable it for the Codex agent, set TYPESAFE_API_KEY and pass --jev-model jev-latest, for example uv run main.py --agent codex --model gpt-5.6-luna --reasoning-effort medium --jev-model jev-latest. Jev then reviews diagnostic tests and submissions using evidence sent to TypeSafe.
More Jev resources like this
217GitHub stars
Standardizes hundreds of public datasets into interchangeable multiple-choice, classification, and token-classification templates, and deterministically recasts them into typed choice, score, and noul decision requests.
Jev Benchmarks & Evaluations#community#jevbest
Community
sileod/tasksourcePython
sileod/tasksourcePython130GitHub stars
Convex's own coding evals, plus a separate multiple-choice decision benchmark in the same repo. Jev is wired into the decision side through OpenRouter's alpha decisions endpoint with a pinned version, while the coding evals still route as chat completions.
Jev Benchmarks & Evaluations#community#jevbest
Community
get-convex/convex-evalsTypeScript
get-convex/convex-evalsTypeScript3GitHub stars
A calibration write-up and scripts for Jev comment and code rules on a Unity project, with ordering.py to compare rule versions on real blocks.
Jev Benchmarks & Evaluations#community#evaluation#code-review
3GitHub stars
A 770-post benchmark that scores Jev, Sonnet 5, GPT-5 nano, and two local models on Reddit verdicts with weighted Brier scores.
Jev Benchmarks & Evaluations#community#python#evaluation
Community
dchristopoulos/jev-aitaPython
dchristopoulos/jev-aitaPython0GitHub stars
Independent pre-registered audit of Jev on Spanish: 19,200 calls over 3,200 paired items on XNLI, PAWS-X, MASSIVE and Belebele. A Spanish state costs 3.0-6.4 accuracy points and roughly doubles ECE on the two hardest tasks, while Spanish instructions change nothing; a CLI reruns the same comparison on your own data.
Jev Benchmarks & Evaluations#community#evaluation#calibration
Community
marcosmartinez/jev-acentoPython
marcosmartinez/jev-acentoPython0GitHub stars
Head-to-head benchmark of fast typed classifiers, TypeSafe Jev, OpenAI Decisions and Cloudflare Clef, against Claude models on a frozen, labeled corpus of 2,321 cases across 31 tasks, measuring accuracy, calibration, p95/p99 latency and registry-priced cost.
Jev Benchmarks & Evaluations#community#benchmarking#calibration
Community
CMaintz/classifier-benchPython
CMaintz/classifier-benchPythonFrom the Jev community
Posts from builders shipping with Jev right now.