User repository on GitHub
Reproducible calibration and selective-risk benchmarks for Jev/TypeSafe decisions in DSPy workflows
- Forks
- 2
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Practices & Patterns
Jev DSPy Lab
Unofficial companion to dspy-typesafeify that records and replays TypeSafe calls to measure calibration, selective risk, confidence-gated abstention, latency, tokens and modeled cost for Jev decisions.
- Stars
- 16
- Language
- Python
- License
- MIT
- Reading time
- 3 min
- Published by
- Community
- Author
- jmanhype
- Last verified
- Sep 2026
About this project
Reproducible calibration, confidence-gating, latency, and modeled-cost benchmarks for Jev / TypeSafe System One decisions used in DSPy workflows. This is not another DSPy fork. It is a companion measurement lab for typesafeainate/dspy-typesafeify and other callers that use TypeSafe decisions inside DSPy-style pipelines. The existing dspy-typesafeify proof of concept is useful and its focused tests pass, but its published benchmark: reports latency and modeled cost but not dataset-level calibration,.
Quoted from the project's README. Read the full README.
Highlights
- The checked 24-case fixture is synthetic and reports 75.0% coverage, 0.667 accuracy among answered, and a Brier score of 0.4844.
- A 24-case live jev-latest recording (response model jev-1.13.0) reports 95.8% coverage, 0.913 accuracy and selective risk 0.087.
- Optional --calibration fits dependency-free Platt scaling only after the confidence gate is declared, and overlapping splits fail.
- Reports include a deterministic exploratory threshold sweep from 0.0 to 1.0; the selected gate stays the confirmatory result.
- Replaying a recording through the offline CLI produces byte-identical JSON and Markdown reports keyed by request hashes.
Quickstart
bash
uv run jev-dspy-benchmark --dataset fixtures/tickets.jsonl --responses fixtures/typesafe_responses.jsonl --output evidence/benchmark --field owner --threshold 0.7Watch out
MIT-licensed. Replay needs no API keys, but recording uses a TypeSafe key; the fixture is synthetic and the project is independent of TypeSafe AI and DSPy.
Questions about this entry
Short answers drawn from the linked project or page.
Both are published, and neither is evidence about Jev quality. The checked fixture is synthetic and deliberately miscalibrated: 24 decisions, 0.667 accuracy among answered, Brier 0.4844. A 24-case live jev-latest recording reports 0.913 and 0.1546. Both are the same 24 synthetic support-ticket routing cases, and the README says so outright.
The live run's confidence gate is 0.700: 23 of 24 decisions were answered and one abstained, for 95.8% coverage and selective risk of 0.087. Coverage and selective risk are both reported so the trade is visible. The synthetic fixture at the same gate abstained 6 of 24, giving 75.0% coverage.
The fixture's 162.0 and 245.1 ms p50 and p95 are properties of the fixture, and the README states explicitly that they are not observed TypeSafe API performance. The live recording shows 191.5 and 405.6 ms for 24 cases, which is closer to reality but still a very small sample on synthetic tickets, not a latency guarantee.
That is what the design is for. Every report includes a deterministic exploratory sweep from 0.0 through 1.0 showing coverage, accuracy and selective risk across gates, and the README is explicit that the selected gate is the confirmatory result and the best sweep row must not be reported as an independent evaluation. Calibration fits only after the gate is declared, and duplicate or overlapping splits fail.
After `uv sync`, run `uv run jev-dspy-benchmark --dataset fixtures/tickets.jsonl --responses fixtures/typesafe_responses.jsonl --output evidence/benchmark --field owner --threshold 0.7`. No API keys are required for the fixture, and it writes benchmark.json, benchmark.md and request_hashes.txt.
Wrap the real TypeSafe client in `RecordingClient`, run the DSPy predictor normally, and it records each canonical request hash, response and latency while skipping duplicates. `ReplayClient` then serves those responses offline, and a missing request raises rather than fabricating a decision.
Adding `--calibration` fits a dependency-free Platt scaling transform on the project's own decision distribution, but only after the selected confidence gate is declared. It writes a separate calibration.json binding Platt parameters, split membership and canonical request hashes, so the refit cannot reuse the gate-selection data.
More Jev resources like this
Independent cross-model benchmark for Jev-class decision models, running 534 frozen cases per complete entrant with scoring code and a four-axis score of accuracy, calibration, latency and cost.
Jev Practices & Patterns#community#python#benchmarks
Community
fstandhartinger/jevbenchPython
fstandhartinger/jevbenchPythonA probability-aware evaluation harness that compares TypeSafe Jev with GLiNER2.5 on zero-shot single-label text classification, measuring calibration, coverage at a fixed error budget, latency, and token cost.
Jev Practices & Patterns#community#python#benchmarks
Community
AbdelStark/jev-benchmarksPython
AbdelStark/jev-benchmarksPythonAn independent calibration study of Jev over three public benchmarks and 900 rule-generated support tickets, publishing every raw Gateway response and the quantization limits of returned probabilities.
Jev Practices & Patterns#community#python#calibration
Community
scienthoon/jev-ood-calibrationPython
scienthoon/jev-ood-calibrationPythonAn independent calibration study of Jev against a frontier fallback on Banking77 and Web of Science, packaged as the janus-decide tool that measures a routing threshold on your own data instead of shipping a default one.
Jev Practices & Patterns#community#python#routing
Community
FirasSX914/JanusPython
FirasSX914/JanusPythonA measurement harness that sends each file snippet to Jev as one Noul question asking whether the content contains a usable secret credential, then scores the returned probability against the expected label.
Jev Practices & Patterns#community#python#security
Community
teyhouse/jev-secret-detectionPython
teyhouse/jev-secret-detectionPythonA pre-registered, independently rescored audit of Jev's calibration and type safety: calibrated on CLINC150 (ECE 0.0204), systematically overconfident on Banking77 (ECE 0.0936), and zero type errors across 8,576 responses.
Jev Practices & Patterns#community#python#calibration
Community
jourdanlabs/assay-001Python
jourdanlabs/assay-001PythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.