Jev-Driven SRE Diagnosis: What Worked and What Failed

It passed 80 of 105 submitted diagnoses (76.2%) across 21 SREGym-Lite faults using jev-1.13.0, with a median diagnosis time of 14.6 seconds and an estimated $0.15 total inference cost.

Two kinds: Jev picking the wrong clue, as when it blamed a raised CPU limit instead of a slow WAF regex in edge request filter CPU saturation, and missing decisive evidence, as when queue depth and retry metrics never reached the pipeline in the hotel reservation retry collapse fault.

A programmatic collector reads Kubernetes objects, events, recent pod logs and resource usage, groups observations by component and summarizes signs of failure. Jev picks the component worth inspecting, then answers whether it is the origin, a downstream victim or unrelated, and selects the numbered evidence item that best shows the mechanism. Jev chooses among supplied options throughout: it generates no commands and writes no report itself.

jev-1.13.0 ran the 21 faults of the September 4 SREGym-Lite cohort five times, 105 submitted diagnoses in total. Scoring was done by gpt-6-astra at high reasoning effort against SREGym's nine-question diagnosis rubric with a 0.70 pass threshold. The article is explicit that this is the historical 21-fault cohort, not the current leaderboard cohort.

16 of 21 faults passed in all five attempts and 5 failed in all five, and 18 of 21 faults scored identically every time. Variance is therefore concentrated in a small number of faults rather than spread across the cohort, which is also why the failure write-up reads as two distinct causes: a wrong clue and missing evidence.

252 Jev calls across the 105 attempts, so 2.4 per attempt, and 3.48 million input tokens. Summed Jev API latency was 0.53 s per attempt against a 14.6 s median end-to-end diagnosis time, with roughly $0.15 of inference cost in total at published TypeSafe pricing. The article puts the pass rate next to GPT-5.6 Sol medium at 77.8% while running about 7x faster and costing about 200x less per diagnosis.

A lot, and it is worth separating the two. On the mutating_webhook_resource_limits_social_network fault the collector found 27 Deployments, spotted that live pods carried a 16Mi limit against a 256Mi template, and matched them to gatekeeper-mutating-webhook-configuration when four other webhooks were present. Jev then chose deployment/nginx-thrift, the admission_webhook category, the origin role, and evidence item E10. It never generated a command or wrote the report, and candidates are examined one at a time.