Jev directory
CommunityJev Benchmarks & Evaluations
Jev Triage — mohammedahmadi1990
Spring Boot and Spring AI benchmark comparing Java keyword rules, Jev 1.13.0, and Claude Haiku 5.5 on warehouse goods-receipt triage, with a 200-remark synthetic dataset and a Python analysis script.
- Stars
- 1
- Reading time
- 2 min
- Published by
- Community
- Added
- Oct 2026
This is one of several Jev listings for the same project. The canonical entry is Jev Triage — boldbug1.
Highlights
- Benchmarks three triage strategies: Java keyword rules, Jev 1.13.0, and Claude Haiku 5.5 via Spring AI.
- Every remark is answered on category, whether goods must be blocked from putaway, and severity.
- Test split of 100 remarks run three times; the dataset holds 200 synthetic remarks, 113 German and 82 English.
- Reports category accuracy of 41.0% for Java rules and 100% for Jev and Claude Haiku 5.5.
- The analysis script reports accuracy, latency, cost, calibration, and draws three charts; MIT licensed.
Questions about this entry
Short answers drawn from the linked project or page.
Create a Python virtual environment, install analysis/requirements.txt, then run python analysis/analyze.py with the benchmark TSV files and --labels src/main/resources/data/remarks.tsv, writing output to analysis/final.
All three approaches answer the same three questions: the category of the problem, whether the goods must be blocked from putaway, and the severity.
remarks.tsv holds 200 synthetic goods-receipt remarks: 113 German, 82 English, and 5 mixed, about 40 per category, split 50/50 into a calibration set and a test set.
On the 100-remark test split, category accuracy was 41.0% for Java rules and 100% for Jev 1.13.0; block accuracy was 66.0% versus 72.0%, and severity 48.0% versus 73.3%.
No. The remarks are synthetic and contain no customer or company data. Labels were drafted with AI assistance and reviewed by hand, and every label change after the first run is documented in Labeling.md.
The dataset is small and synthetic, so differences of a few points are noise; the criteria v2 wording and the hazmat rule were designed after seeing first-run errors, and results reflect Jev 1.13.0 and Claude Haiku 5.5 in October 2026.
The repository is MIT licensed and maintained by Mohammed Ahmadi. It is the companion code for his article on testing Jev with Spring Boot.
More Jev resources like this
Independent experiment scoring 400 real job postings against 30 fictional résumés with Jev, producing 12,000 evaluations in about 197 seconds for roughly $0.17, with a 100-posting answer key and an analysis of what the model got wrong.
Jev Benchmarks & Evaluations#github#madewithjev#evaluation
Communitykaucap08/jev-job-matching
A measurement lab for TypeSafe AI's Jev, run against api.typesafe.ai/v1/systemone with model jev-latest. It records baseline confidence scores, position-bias controls, atomic-question failures, and a 24-README semantic triage sweep, with committed data and analysis scripts.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
CommunitySuperInstance/jev-lab
Hugging Face dataset of 130k rows pairing RAG claim-verification and response-quality questions with Jev-1.13.0 predictions, gold labels, confidences, token counts, latency, and cost. Derived from wandb/RAGTruth-processed.
Jev Benchmarks & Evaluations#evaluation#classification#benchmark
Community
A fast, typed Python framework from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models: Jev by default, Microsoft Decision-1 on OpenRouter or OpenAI's Decisions API through the same backend. It returns typed scores, thresholds and status, with optional human-label calibration.
Jev Benchmarks & Evaluations#community#github#python
Community
trustifai/typed_evalsPython
trustifai/typed_evalsPythonThe code and data behind ProbabilityBench: text-based simulated worlds with exact class probabilities, so a decision model's reported probabilities can be compared directly with ground truth. An arXiv paper and a Hugging Face dataset accompany it.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
ishida-lab/probabilitybenchPython
ishida-lab/probabilitybenchPythonBenchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.