Question 1

What is MedHallu and what does Jev check?

Accepted Answer

MedHallu is a Stanford MedHELM benchmark: given a PubMed abstract, a question and a proposed answer, decide whether the answer is hallucinated. Jev answers one Noul question asking whether the study's authors would say the answer misrepresents their findings.

Question 2

How do the LLM call savings work?

Accepted Answer

On the 37% of items where Jev was at least 90% confident, all four comparison LLMs agreed with Jev anyway, so letting Jev settle those and sending the rest to an LLM kept accuracy with 37% fewer calls.

Question 3

Where does Jev actually land on accuracy?

Accepted Answer

Fourth of the hosted set at 92.9% against Gemini 3.5 Flash Lite at 95.0%, GPT-5.6 Luna at 95.1% and Gemini 3.8 Flash at 94.8%, about 2 points behind each on exact McNemar with Holm p of 0.012 to 0.016, while Claude Haiku 4.5 at 92.4% is level. Against Stanford's 13 published models it is level with Claude 3.5 Sonnet and significantly ahead of 11.

Question 4

How fast and cheap is it in the timed run?

Accepted Answer

204 ms median, 292 ms at the 95th percentile and USD 0.03 per 1,000 checks, which is 2.6 to 6.5 times faster and 5 to 35 times cheaper than the four hosted LLMs. Re-timed in one alternating session the medians were 228 ms for Jev against 703 to 1,460 ms, so the gap is not an artifact of run order.

Question 5

Can I treat the probability as a literal odds estimate?

Accepted Answer

No. It ranks well, with AUROC 0.974 and 99.5% accuracy on the 37% of items where it is at least 90% confident, but of the answers it put near 14%, 2% were hallucinated, and of those near 65%, 43% were. The calibration file is marked exploratory and the advice is to set cut-offs on your own data.

Question 6

What does the benchmark not test, and how do I reproduce it?

Accepted Answer

With the abstract removed every model still scores 66% to 83% against a 52% floor, so MedHallu does not only test source checking, and Jev's question and threshold were tuned on 1,000 separate dev items while the LLMs were not. Reproduce with uv run datasets/medhallu/prepare.py then uv run score.py runs/medhallu/test-v2 --threshold 0.65 --ref jev-1.13 --xlsx; scoring needs no key.

Question 7

What are the licensing terms for the code and the data?

Accepted Answer

The code and docs are MIT. Data keeps its own terms: MedHallu items are MIT from UTAustin-AIHealth/MedHallu, built on PubMedQA, the ICD-10 notes are CC-BY-4.0 synthetic notes with no patient data, and Stanford's per-item MedHELM predictions are not re-hosted but downloaded from their public bucket, so the scored files embedding them are git-ignored.

Jev Medhallu Benchmark

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev community