Question 1

Is 56.6% on a five-option multiple-choice exam actually any good?

Accepted Answer

It is above the 20.0% random baseline and level with a 120B-parameter mixture-of-experts model on the same 182 valid items, but the area split is the informative part: 78.0% in Linguagens and 73.3% in Humanas against 46.5% in Natureza and 25.0% in Matemática. The README attributes the quantitative collapse to the absence of a calculator or code interpreter and to plain-text maths without advanced LaTeX formatting.

Question 2

Does letting the model reject an item just inflate the score by discarding the hard ones?

Accepted Answer

Partly, and the study reports both sides. With the escape option, accuracy is 78.8% on the 89 items Jev answered itself; forced to guess on the items it had rejected, accuracy drops to 35.5%, still above the random line. The escape also changes calibration: ECE reads 0.275 when a rejection counts as an error and 0.078 when only the exam alternatives are scored, so the two numbers describe different questions.

Question 3

How do I check the calibration claim instead of taking the ECE on trust?

Accepted Answer

The README gives the band-level evidence behind the aggregate: in the 80 to 100% declared confidence band, Jev's empirical accuracy is 89.1% while Nemotron 120B falls to 62.9%, and in the 40 to 60% band Jev lands at 50.0%. Raw per-run logs are written to runs/*_results.jsonl and the metrics come from calibration_metrics.py, which also computes multiclass Brier score, with the consolidated output in docs/data/benchmark_summary.json.

Question 4

Where does the exam data live, and what is actually in the repository?

Accepted Answer

The tabular data is not committed. It is published as a Kaggle dataset to keep the repository light, with separate official answer keys and question files for day one (Linguagens e Humanas) and day two (Natureza e Matemática). The repository holds the experimental protocol, the calibration engine and a GitHub Pages dashboard that reads the consolidated JSON directly. Fetch the dataset separately before trying to reproduce anything.

Question 5

How large is the ENEM 2025 evaluation set?

Accepted Answer

182 valid items out of 185 consolidated, with 3 annulled. The questions and answer keys are hosted publicly on Kaggle as CSVs covering both exam days, including a languages-option field for day one.

Question 6

What exactly did the escape option do to accuracy?

Accepted Answer

Jev could answer A to E or return INVALIDO when it judged the alternatives insufficient. On items it accepted, accuracy was 78.8%; on items it had rejected, forcing a choice in round two dropped accuracy to 35.5%, still above the 20% random baseline, which the study reads as real uncertainty detection.

Question 7

How consistent was Jev between the two rounds?

Accepted Answer

On the 86 items where Jev made a firm choice with the escape option available, it repeated exactly the same alternative on 84, an intra-model agreement of 97.7%. That stability is measured across the two rounds on the same items.

Jev No Enem

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev guides

From the Jev community