Jev directory
CommunityJev Benchmarks & Evaluations
Jev Job Matching
Independent experiment scoring 400 real job postings against 30 fictional résumés with Jev, producing 12,000 evaluations in about 197 seconds for roughly $0.17, with a 100-posting answer key and an analysis of what the model got wrong.
- Stars
- 1
- Reading time
- 2 min
- Published by
- Community
- Added
- Oct 2026
Highlights
- 400 real job postings from a public Kaggle dataset scored against 30 fictional résumés
- 12,000 candidate-job evaluations completed in about 197 seconds for roughly US$0.17
- Top 3 matched the answer key on 96 of 100 hand-checked postings
- Top 1 was 45/100 with the 1.5 cutoff and 60/86 without it
- All 14 postings with no suitable candidate scored below the cutoff and were called correctly
Questions about this entry
Short answers drawn from the linked project or page.
Run python avaliar.py. It compares resultados.csv against gabarito_claude.csv and recomputes the top-1, top-3, and none-detection numbers locally without contacting the API.
About US$0.17 in TypeSafe API credits. The run used 3,988,299 input tokens across 400 postings. You need your own key in a .env file, and the author paid for all calls with personal credits.
Each job posting description plus the 30 fictional résumés. The trap notes section of curriculos_ficticios.md, which describes expected behavior, is deliberately not sent to the model.
The CSV files and the fictional résumés are CC BY-SA 4.0. The Python scripts are MIT. The underlying postings come from the LinkedIn Job Postings dataset on Kaggle, also CC BY-SA 4.0.
It was tested with Python 3.14. Setup is a virtual environment plus pip install -r requirements.txt. A TypeSafe API key is only needed to re-run rodar_jev.py.
Most misses are seniority mix-ups: 16 of 26 top-1 misses without the cutoff. The 1.5 cutoff was also too high, since the median best score across 400 postings was 1.31 and 225 postings fell below it.
The answer key was labeled by Claude and only spot-checked by the author on a sample, so it is not fully human-labeled. Kaggle updates the source dataset, and model outputs can differ between runs and model versions.
More Jev resources like this
Spring Boot and Spring AI benchmark comparing Java keyword rules, Jev 1.13.0, and Claude Haiku 5.5 on warehouse goods-receipt triage, with a 200-remark synthetic dataset and a Python analysis script.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
Communitymohammedahmadi1990/jev-triage
A measurement lab for TypeSafe AI's Jev, run against api.typesafe.ai/v1/systemone with model jev-latest. It records baseline confidence scores, position-bias controls, atomic-question failures, and a 24-README semantic triage sweep, with committed data and analysis scripts.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
CommunitySuperInstance/jev-lab
A Python benchmark that evaluates LLMs' reasoning and instruction-following by making them play chess against a random player or the Komodo Dragon engine in multi-turn dialogs, separating chess skill from protocol durability.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
maxim-saplin/llm_chessPython
maxim-saplin/llm_chessPythonA timestamped benchmark and trading arena that replays resolved prediction markets: models commit to probabilities using only information available at the time, and realized P&L is the headline metric with Brier, log-loss and ECE as checks.
Jev Benchmarks & Evaluations#community#github#python
Community
ruyianry/JevGymPython
ruyianry/JevGymPythonGTM workflow experiments with Jev from deepdave98: weekly datasets, prompts, and benchmark results that test Jev against Claude on lead triage and routing, deal risk, reply triage, MEDDPICC from call summaries, duplicate accounts, and closed-lost reactivation.
Jev Benchmarks & Evaluations#community#github#gtm
Community
deepdave98/jev-playgroundPython
deepdave98/jev-playgroundPythonA fast, typed Python framework from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models: Jev by default, Microsoft Decision-1 on OpenRouter or OpenAI's Decisions API through the same backend. It returns typed scores, thresholds and status, with optional human-label calibration.
Jev Benchmarks & Evaluations#community#github#python
Community
trustifai/typed_evalsPython
trustifai/typed_evalsPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.