CommunityJev Benchmarks & Evaluations
PT-BR Typed Decisions Benchmark
A native Brazilian Portuguese typed-decisions benchmark: seven choice and noul tasks over legislative, legal, scientific, and fact-checking text, with a tuned 4B model at 72.9% mean.
- Format
- Docs
- Published by
- Community
- Author
- Felhen
- Last verified
- Oct 2026
Works forJev for Evaluation
- Capabilities
- Open source
Highlights
- No machine-translated rows: every task is built from native pt-BR sources with human labels or official fields.
- Seven tasks, from Câmara bill themes to OLID-BR offense detection, each capped at 1,000 test rows.
- Tuned Saracura 4B reaches 72.9% mean against a 27.6% majority-class floor, with untuned 4B decision models at 59.0–64.1%.
- A quarterly fresh probe uses 438 Câmara bills submitted from 01/05/2026 to track contamination.
Watch out
No score tasks exist — the card found no pt-BR source with grades and text under a license allowing commercial redistribution. Some tasks are imbalanced (84% of one OLID test split is offensive), so balanced accuracy matters, and public sources make contamination likely.
Questions about this entry
Short answers drawn from the linked project or page.
No. The card says it is built from native sources — OLID-BR, FACTCK.BR, FaQuAD-NLI, SciELO, JurisTCU, and Câmara open data — explicitly without automatic translation, though class-option descriptions were generated by Qwen3.8-27B and reviewed.
The dataset ships its own leaderboard: tuned Saracura 4B at 72.9% mean, untuned 4B decision models at 59.0–64.1%, TF-IDF at 64.0%, and a 27.6% majority-class floor. A 400-example adaptation track takes Kev 4B from 62.2% to 69.9%.
OLID-BR, FACTCK.BR, FaQuAD-NLI, SciELO, JurisTCU and Camara open data. Every task uses native pt-BR sources with human labels or official fields and no automatic translation, though class-option descriptions were generated by Qwen3.8-27B and reviewed.
No. The card found no pt-BR source with grades and text under a license allowing commercial redistribution, so the benchmark carries only choice and noul tasks.
Seven tasks, each capped at 1,000 test rows. The released splits carry roughly 14,000 train rows, 300 dev rows and 599 test rows, across subsets such as camara_tema, olid_ofensivo and scielo_area.
A quarterly fresh probe uses 438 Camara bills submitted from 01/05/2026. The card notes that public sources make contamination likely, which is why the probe exists.
Balanced accuracy, because some tasks are imbalanced: 84% of one OLID test split is offensive. The leaderboard reports tuned Saracura 4B at 72.9% mean and TF-IDF at 64.0% against a 27.6% majority-class floor.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
0GitHub stars
Independent pre-registered audit of Jev on Spanish: 19,200 calls over 3,200 paired items on XNLI, PAWS-X, MASSIVE and Belebele. A Spanish state costs 3.0-6.4 accuracy points and roughly doubles ECE on the two hardest tasks, while Spanish instructions change nothing; a CLI reruns the same comparison on your own data.
Jev Benchmarks & Evaluations#community#evaluation#calibration
Community
marcosmartinez/jev-acentoPython
marcosmartinez/jev-acentoPythonThe 400-case, 2,000-decision benchmark open models train and report against: four workflows, state plus typed questions, and gold distributions averaged from three teacher samples.
Jev Benchmarks & EvaluationsDocs#community#benchmarks#evaluation
Community
2.5 million typed decisions recast from 670 sources: choice, score, and noul rows with per-row licenses, soft annotator targets, and Jev-audited mix weights.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
572,000 procedurally generated decision problems where labels are computed exactly from rules in the state: 13 task families, 4–60 options, and difficulty calibrated against Jev 1.13.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
An independent 2,018-item answer-verifier benchmark: the same items and labels for every entrant, each run on its own maker's code, with Jev, ZTC-Judge, Darwin, and open models ranked by weighted AUC.
Jev Benchmarks & EvaluationsPlayground#community#evaluation#benchmarks
Community
The System One Mosaic Benchmark leaderboard: about 100 decision models compared across 100+ specialized tasks, with results regenerated from a public dataset rather than estimated.
Jev Benchmarks & EvaluationsPlayground#community#evaluation#benchmarks
Community
From the Jev guides
Original write-ups on what this entry is for.
Base decision checkpoints sit near chance on purpose. The typed-decisions loop, the corpora behind the students, and the temperature fit that turns probabilities into thresholds.
Jev Practices & Patterns#fine-tuning#calibration#open-models
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.