Jev directory
CommunityJev Benchmarks & Evaluations
TypeSafe Experiment
Pkl-based divergence tracker for a small TypeSafe AI API experiment. Four messages are scored by the is_urgent question on jev-latest and compared with a pinned baseline, using sanity bands and tolerances to flag behaviour drift.
- Stars
- 0
- Reading time
- 2 min
- Published by
- Community
- Added
- Oct 2026
Highlights
- Scores four messages with the is_urgent question on jev-latest and compares them to a pinned baseline.
- pkl/TypesafeExperiment.pkl holds the endpoint, messages, sanity bands, pinned baseline, and tolerances.
- scripts/capture.sh is the only component that touches the network and writes build/typesafe-observed.json.
- Baseline recorded 2026-10-25 on jev-1.14.0; mean drift beyond a 0.05 tolerance counts as divergence.
- Requires pkl, jq, curl, and pass-cli logged in with access to a stored TYPESAFE_API_KEY; the calls are paid.
Questions about this entry
Short answers drawn from the linked project or page.
Run bash scripts/capture.sh to capture observed scores, then run pkl test pkl/TypesafeExperiment.test.pkl. The environment needs pkl, jq, curl, and pass-cli available, and TYPESAFE_RUNS (default 3) controls how many repetitions each capture performs.
The script reads the key from pass://infra/typesafe.ai/TYPESAFE_API_KEY through pass-cli. The key is read inside the script and passed to curl on stdin; it is never written into build/ and never printed.
Yes. The digest notes that the API calls are paid, and TYPESAFE_RUNS, defaulting to 3, sets how many repetitions each capture makes, so higher repetition counts mean more billed calls.
Divergence is flagged when the resolved model differs from baselineModel, when a mean score leaves its sanity band, when the urgency ranking changes, when a mean drifts more than the 0.05 tolerance, or when spread across one capture's runs exceeds the 0.10 spreadTolerance.
Update baselineModel and baselineMean in the Pkl spec to reflect the new intended behaviour, then run pkl test --overwrite pkl/TypesafeExperiment.test.pkl to snapshot the new expectations.
The quickstart docs show 0.99 on jev-1.13.0 for the api-500s sentence, while roughly 0.78 was observed. That known gap is retained in the spec as docsExample rather than treated as drift.
It covers only four messages and one question against jev-latest, so it detects drift relative to its own pinned baseline rather than measuring broad calibration or general model quality. The resolved-model check fails on purpose, so a model change must be reviewed and re-pinned.
More Jev resources like this
Spring Boot and Spring AI benchmark comparing Java keyword rules, Jev 1.13.0, and Claude Haiku 5.5 on warehouse goods-receipt triage, with a 200-remark synthetic dataset and a Python analysis script.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
Communitymohammedahmadi1990/jev-triage
Independent audit repository testing whether Jev's probabilities are calibrated on EIA weekly petroleum reports, comparing Jev via LangChain TypeSafeClassifier against an LLM-as-classifier, rules, and climatology baselines.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
Communityn9omi/jev-calibration-audit
Evaluation of MedJev, an open System One model that runs locally on a laptop, across ten medical benchmarks for knowledge, diagnosis, and patient-level prediction on MIMIC IV and MIMIC IV ED, with a two-parameter recalibration method and a released evaluation harness.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
A measurement lab for TypeSafe AI's Jev, run against api.typesafe.ai/v1/systemone with model jev-latest. It records baseline confidence scores, position-bias controls, atomic-question failures, and a 24-README semantic triage sweep, with committed data and analysis scripts.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
CommunitySuperInstance/jev-lab
A fast, typed Python framework from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models: Jev by default, Microsoft Decision-1 on OpenRouter or OpenAI's Decisions API through the same backend. It returns typed scores, thresholds and status, with optional human-label calibration.
Jev Benchmarks & Evaluations#community#github#python
Community
trustifai/typed_evalsPython
trustifai/typed_evalsPythonThe code and data behind ProbabilityBench: text-based simulated worlds with exact class probabilities, so a decision model's reported probabilities can be compared directly with ground truth. An arXiv paper and a Hugging Face dataset accompany it.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
ishida-lab/probabilitybenchPython
ishida-lab/probabilitybenchPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.