Jev directory
CommunityJev Benchmarks & Evaluations
Clause Classification with Jev
Python project that measures how much CUAD license-agreement text Jev can bin before an expensive model reads it, reporting bin rates, lost lawyer-marked provisions, and estimated cost savings against a Claude Sonnet baseline.
- Stars
- 0
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Bins 77.3% of Clauses at the 0.5 cutoff across 53 CUAD license agreements and 3,246 Clauses.
- Loses 44 of 429 lawyer-marked provisions (10.3%) inside those binned Clauses.
- Estimates about $24 of $26.45 saved (91.4%) versus Claude Sonnet 5.5 at list prices.
- Real run used jev-1.14.0: 3,246 fan-out calls, 22,722 Pairs, 179 s, $0.085 as Jev reported.
- CUAD is not bundled; point CUAD_DIR at CUAD_v1 and run the CLI with uv.
Questions about this entry
Short answers drawn from the linked project or page.
Run uv sync --extra dev, set CUAD_DIR to your CUAD_v1 directory, then uv run jev-cascade --out out. The default uses a signal-free stub; add --jev real with JEV_API_KEY set to call TypeSafe's Jev.
It measures how many Clauses Jev fully bins at a 0.5 cutoff, how many lawyer-marked provisions are lost inside binned Clauses, and the estimated cost saved by not sending those Clauses to an expensive model.
The dataset is not in the repo. It is expected at ~/personal_coding/playground/cuad/CUAD_v1/, and CUAD_DIR can point elsewhere. Tests that need CUAD skip when it is absent.
The full run is 3,246 fan-out calls covering 22,722 Pairs, at most about $0.27 at the published price. The completed run reported $0.085 in Jev cost and took 179 seconds on jev-1.14.0.
The real client reads the key from JEV_API_KEY in the environment, never from the repo; a git-ignored .env can be loaded into the shell first. Rebuilding with --from-cache needs no key and makes no calls.
Yes. --from-cache rebuilds the report from the JSONL P(yes) cache with no key and no calls. It is reproducible from the cache, not from scratch: real Jev moved by up to 0.04 between runs on the same Pair.
At the 0.5 cutoff, 44 of 429 lawyer-marked provisions (10.3%) are lost inside binned Clauses. Five Rules catch 92 to 94% of lawyer-marked Clauses, Minimum Commitment catches 60% and Change Of Control 75%.
More Jev resources like this
Independent experiment scoring 400 real job postings against 30 fictional résumés with Jev, producing 12,000 evaluations in about 197 seconds for roughly $0.17, with a 100-posting answer key and an analysis of what the model got wrong.
Jev Benchmarks & Evaluations#github#madewithjev#evaluation
Communitykaucap08/jev-job-matching
Spring Boot and Spring AI benchmark comparing Java keyword rules, Jev 1.13.0, and Claude Haiku 5.5 on warehouse goods-receipt triage, with a 200-remark synthetic dataset and a Python analysis script.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
Communitymohammedahmadi1990/jev-triage
Hugging Face dataset of teacher judgments for ROR affiliation point matching, named for Jev distillation. Each row pairs an affiliation text with ROR identifiers and calibrated probabilities, across 200k rows in train and validation splits.
Jev Benchmarks & Evaluations#calibration#evaluation#classification
Community
Portuguese-language technical study of Jev, TypeSafe AI's System One decision model, benchmarked against a structured-output LLM baseline. Ships a Python 3.11 harness, a 45-task synthetic eval set, and a runner that records decisions, confidence, latency, tokens, and estimated cost.
Jev Benchmarks & Evaluations#github#evaluation#calibration
CommunityGabriel-Abi-Acl/jev-study
Independent audit repository testing whether Jev's probabilities are calibrated on EIA weekly petroleum reports, comparing Jev via LangChain TypeSafeClassifier against an LLM-as-classifier, rules, and climatology baselines.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
Communityn9omi/jev-calibration-audit
A measurement lab for TypeSafe AI's Jev, run against api.typesafe.ai/v1/systemone with model jev-latest. It records baseline confidence scores, position-bias controls, atomic-question failures, and a 24-README semantic triage sweep, with committed data and analysis scripts.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
CommunitySuperInstance/jev-lab
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.