Should Jev instructions be written in Spanish or English?

The audit found no detectable accuracy or calibration difference from translating instructions and criteria into Spanish. Since a Spanish state still costs accuracy versus English and Spanish wording cost 6.2% more input tokens on MASSIVE, the practical advice is to keep instructions in English.

Can I run this comparison on my own data?

Yes. acento compare takes a JSONL file with id, state and gold per question plus two or more question-set versions, and outputs paired deltas for accuracy, ECE against a noise floor, coverage at thresholds and tokens. No parallel English data is needed.

How much did the audit cost, and how big was it?

Run 20260921-es-v1 is 19,200 calls over 3,200 paired items against the version-pinned model jev-1.13.0, which came to USD 0.58 with 0 errors. Every figure in the README is regenerated from results.json by make reproduce rather than typed by hand.

How do I know the model's own randomness is not producing those gaps?

The repo checked that first. Two passes sent byte-identical requests, producing flip rates of 0.2-2.1% and Cohen's kappa of 0.95 or above, and accuracy jitter between passes of 0.000-0.005 against reported deltas of 0.030-0.064, so the effect is an order of magnitude above the noise.

What does a Spanish state cost me when I automate above a threshold?

Coverage drops and the accuracy of what you do automate drops too. Automating at p_max of 0.9 or above covers 72.2% of XNLI in English but only 63.4% in Spanish, and those automated items fall from 0.938 to 0.904 accuracy, with ECE roughly doubling on XNLI and PAWS-X.

How do I install it, and which provider should I point it at?

uv tool install jev-acento, or pipx install jev-acento. acento run --provider typesafe is preferred for the audit because it is the path that reports which model version answered, while --provider gateway goes through Vercel AI Gateway so the work is reproducible without an approved account. acento smoke checks a provider in 3 calls.

Is the pre-registration actually frozen yet?

Not yet, and the tooling refuses to pretend otherwise. PREREG.md is a draft awaiting review of the Spanish wordings, so make check-prereg correctly reports that nothing is registered and make run refuses to write to runs/. make verify is the full pre-publication gate.

CommunityJev Benchmarks & Evaluations

Jev Acento

Independent pre-registered audit of Jev on Spanish: 19,200 calls over 3,200 paired items on XNLI, PAWS-X, MASSIVE and Belebele. A Spanish state costs 3.0-6.4 accuracy points and roughly doubles ECE on the two hardest tasks, while Spanish instructions change nothing; a CLI reruns the same comparison on your own data.

See highlights

Open github.com

Stars: 0
Language: Python
License: MIT
Published by: Community
Last verified: Oct 2026

marcosmartinez/jev-acento

User repository on GitHub

View on GitHub

¿Jev entiende tu acento? Pre-registered audit of TypeSafe AI's Jev on Spanish — accuracy, calibration and token cost — plus a CLI to run the same comparison on your own labelled data.

Forks: 1
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.Homepage

About this project

An independent, reproducible audit of Jev — TypeSafe AI's "System One" evaluation model — on Spanish, plus a CLI that lets anyone run the same comparison on their own labelled data. Run 20260921-es-v1 — 19,200 calls, 3,200 paired items, model jev-1.13.0 (version-pinned), USD 0.58, 0 errors. Every number below comes from results.json via make reproduce; none is typed by hand.

Quoted from the project's README. Read the full README.

Tagscommunity evaluation calibration multilingual

Works forJev for Evaluation Jev for Classification

Used inJev for Software & DevTools

Languages: Python
Platforms: CLI
Capabilities: Open sourceRequires API key

Highlights

Run 20260921-es-v1: 19,200 calls, version-pinned model jev-1.13.0, USD 0.58 and 0 errors.
XNLI accuracy drops 6.4 points with a Spanish state; Spanish instructions show no detectable difference.
Spanish text costs 17-38% more input tokens than the same content in English.
acento compare reruns the audit on your own JSONL labelled data; a dry-run example needs no key.

Watch out

MIT licensed for code and derived rows. Each cell uses a single wording, four academic benchmarks may not transfer to real workloads, and the pre-registration was still a draft awaiting native-speaker review when checked.

Questions about this entry

Short answers drawn from the linked project or page.

Related terms

Glossary definitions related to this entry.

More Jev resources like this

Jev on r/AmItheAsshole

A 770-post benchmark that scores Jev, Sonnet 5, GPT-5 nano, and two local models on Reddit verdicts with weighted Brier scores.

Jev Benchmarks & Evaluations#community#python#evaluation

Community

dchristopoulos/jev-aitaPython

jev-harness-lab

Evaluation scripts, MCP tools, and a skill router built to measure where jev-1.13.0 fits in agent harnesses, with raw results under eval/.

Jev Benchmarks & Evaluations#community#evaluation#agents

Community

Aitejiu/jev-harness-labPython

classifier-bench

Head-to-head benchmark of fast typed classifiers, TypeSafe Jev, OpenAI Decisions and Cloudflare Clef, against Claude models on a frozen, labeled corpus of 2,321 cases across 31 tasks, measuring accuracy, calibration, p95/p99 latency and registry-priced cost.

Jev Benchmarks & Evaluations#community#benchmarking#calibration

Community

CMaintz/classifier-benchPython

LangWatch

4.9k

LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.

Jev Benchmarks & Evaluations#community#typescript#evaluation

Community

langwatch/langwatchTypeScript

hev reranker

A BEIR reranking recipe that asks up to 30 parallel Noul relevance questions in one Jev state, with RESULTS.md against cross-encoders and hosted rerankers.

Jev Benchmarks & Evaluations#community#evaluation#python

Community

hev/rerankerPython

jev vs. open alternatives

Notebook benchmark that compares Jev with open models and specialised tools on five document-pipeline chores: language detection (Choice(12)), page orientation (Choice(4)), RVL-CDIP classification (Choice(16)), bundle splitting and parse-tier triage, with liteparse doing all PDF rendering and OCR work.

Jev Benchmarks & Evaluations#community#evaluation#document-processing

Community

run-llama/jev_vs_ossJupyter Notebook

From the Jev guides

Original write-ups on what this entry is for.

Fine-Tuning and Calibration: Making an Open Checkpoint Yours

Oct 9, 2026

Base decision checkpoints sit near chance on purpose. The typed-decisions loop, the corpora behind the students, and the temperature fit that turns probabilities into thresholds.

Jev Practices & Patterns#fine-tuning#calibration#open-models

Read article

Measuring Jev: What Independent Benchmarks Actually Found

Sep 22, 2026

TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.

Jev Practices & Patterns#benchmarks#evaluation#calibration

Read article

Back to all resources

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai