Are the accuracy numbers independent, or did the authors grade their own model?

The gold labels come from blind adjudication of the disagreements with Claude Opus 5, and the decisions are committed in data/adjudicacao/, with raw per-model answers in data/anotacoes/ as JSONL. That is a stronger setup than self-labelling, though the process is manual: the reference script only lists divergences and an audit for a human to resolve. Note also that the 120 sentences are not in the repository, so you cannot re-grade them without recollecting.

Does Jev win, or is the cheap model just good enough?

On accuracy it does not win. Gemini 3.8 Flash scored 98.8 percent against Jev's 96.6 percent, roughly 26 times the cost per thousand sentences; GPT-5.6 Luna scored 96.8 percent at US$1.64 per thousand. The case for Jev in this table is latency and unit cost: 0.32 s median against 2.81 s and 3.49 s, and US$0.25 per thousand sentences. Whether that trade is worth two points of accuracy depends entirely on how often you annotate.

Are the three models actually being asked the same thing?

Yes by construction, which is the point of the schema module: src/jevtest/schema.py is the single source of the annotation questions for all three providers, so nothing provider-specific can drift into the prompts. What differs is the transport: src/jevtest/clientes.py makes the asynchronous calls and measures latency, with Gemini and GPT driven through structured output while Jev answers typed questions with probabilities.

Does the study look at calibration, or only at whether answers match the labels?

Calibration is in scope: src/jevtest/avaliacao.py computes quality, time, cost and calibration metrics together. The summary table the README leads with reports only accuracy, median latency and cost per thousand sentences, so the calibration figures live in the rendered report and the JSON output rather than in the headline numbers, which is worth reading before trusting the confidence values on any single sentence.

What task and dataset does the LabDados study use?

Annotating 12 variables across 120 civil sentences from the TJSP collected with juscraper, comparing Jev with Gemini 3.8 Flash and GPT-5.6 Luna used with structured output.

What keys do I need to reproduce the annotation experiment?

JEV_API_KEY or TYPESAFE_API_KEY for Jev, GEMINI_API_KEY for Gemini and OPENAI_API_KEY for the OpenAI models; the .env paths are documented in src/jevtest/config.py.

How is the study's gold standard built?

Divergences go through a blind adjudication handled by Claude Opus 5, with decisions recorded under data/adjudicacao; the resulting gold standard is data/gold.parquet.

CommunityJev Practices & Patterns

Jev Anotacao Sentencas

LabDados FGV Direito SP experiment that annotates 12 variables across 120 TJSP civil sentences and compares jev-latest with Gemini 3.8 Flash and GPT-5.6 Luna on accuracy, median latency and cost per thousand sentences.

See highlights

Open github.com

Stars: 0
Language: Python
Published by: Community
Author: lab-dados
Last verified: Oct 2026

lab-dados/jev-anotacao-sentencas

Organization repository on GitHub

View on GitHub

Jev (TypeSafe) vs. Gemini 3.8 Flash vs. GPT-5.6 Luna na anotação estruturada de sentenças do TJSP: qualidade, tempo e custo

Forks: 0
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.Homepage

About this project

Experimento do LabDados FGV Direito SP. Compara o Jev (TypeSafe), um modelo "System One" que só responde perguntas tipadas com probabilidades calibradas, com o Gemini 3.8 Flash e o GPT-5.6 Luna usados com saída estruturada. A tarefa é anotar 12 variáveis em 120 sentenças cíveis do TJSP coletadas com o juscraper. 📄 Relatório: · PDF (Typst) · fonte .typ. src/jevtest/schema.py: esquema de anotação, fonte única das perguntas para os três provedores. src/jevtest/clientes.py: chamadas assíncronas e medição de latência.

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import awesomejev

Works forJev for Classification

Used inJev for Software & DevTools

Languages: Python

Highlights

Jev reached 96.6 percent accuracy at 0.32 s median latency and about US$0.25 per thousand sentences, against Gemini 3.8 Flash at 98.8 percent and 2.81 s.
A single schema module is the source of the questions for all three providers, so the comparison is over one question set rather than three prompt dialects.
The gold standard comes from blind adjudication of the disagreements with Claude Opus 5, with the decisions recorded in data/adjudicacao/.
Full sentence text is not redistributed under LGPD art. 6 III, and scripts/01_coletar.py re-runs the collection from the source instead.

Watch out

The headline table is three rows on 120 sentences from a single Brazilian court, which is small enough that a couple of points of accuracy sit inside the noise; treat the comparison as directional rather than settled. Jev is the cheapest and by far the fastest of the three here, but Gemini 3.8 Flash is the most accurate at 98.8 percent, and the README does not argue otherwise. Reproducing the run needs three provider keys at once, JEV_API_KEY or TYPESAFE_API_KEY plus GEMINI_API_KEY and OPENAI_API_KEY, and every step past collection is a paid call across providers, so the experiment is not free to check. Because the sentence text is withheld for LGPD reasons, you cannot re-score the same 120 items from a clone; you have to recollect. The adjudication step is manual: the reference script lists divergences and an audit, then someone writes data/adjudicacao/decisoes.csv before the gold file is regenerated. Numbers quoted here are the project's own run on its own sample. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-on-a-laptop

Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.