Question 1

Are these probabilities real, or parsed out of text the model generated?

Accepted Answer

They come from the model itself. A mask is placed beside each candidate and the masked diffusion model's shared Yes/No weights are read directly, so choice, boolean and ordered-score answers are a native readout rather than probabilities recovered from text the model wrote and then parsed. The README is explicit about that distinction, and the HTTP service returns the distribution keyed by your original candidate strings. Whether those numbers are calibrated is a separate question, answered separately in the results docs.

Question 2

What could contaminate the benchmark numbers?

Accepted Answer

Several things, and the repository documents each instead of glossing it. Preparation replays a frozen audited membership selection rather than re-running the full private exposure audit, so it is relative only to known training and dev exposure; pretraining overlap is unknown and the public items have already been observed. The model comparison drops sources confirmed seen during training and keeps the rest, but retained items still have unconfirmed source overlap, which the docs explicitly call out as not a guarantee of contamination-free data. Read the data card before quoting a number.

Question 3

What does running it require, and what is the sharp edge?

Accepted Answer

Python 3.10, an NVIDIA GPU supporting BF16, and a driver compatible with CUDA 12.1; memory needs move with input length and batch size, so the instructions tell you to lower the batch and record the change when you hit OOM. The sharp edge is the model loader: it runs with `trust_remote_code=True`, which executes custom code inside the checkpoint, so read it before trusting the source. And validation reused an existing dependency environment — a from-scratch install of every dependency was not verified, and the lock file is explicitly incomplete.

Question 4

Can I call this from an application, or is it a paper artifact?

Accepted Answer

There is a local HTTP service, not only scripts. After evaluation a serve command starts on localhost and answers a health check plus `/v1/systemone`, returning the chosen answer and a probability distribution keyed by your original candidate strings. It is built on the Python standard library, so there is no FastAPI to install, and a protocol document defines the three question formats and their limits. It binds to the loopback interface only, so exposing it publicly is your problem to solve.

Question 5

How big is the benchmark, and how is it prepared?

Accepted Answer

The core benchmark reconstructs 17 suites and 6,744 decisions from 14 fixed source entries, not from models. A prepare script fetches those entries under their own terms and writes to an output directory that must not already exist, leaving raw data in a cache outside the repo. A --smoke --smoke-limit 3 flag runs the first three rows of each suite into a separate output directory.

Question 6

What do the calibration numbers look like, and does calibration help?

Accepted Answer

The S1 global temperature calibration fits T=1.87397 over six suites and reports average ECE15 10.33%, NLL 0.8808 and Brier 0.4637 on the same denominator, but the README notes some tasks degrade under it and the HTTP service and default evaluation still use T=1. A Dream-7B zero-shot run reports average ACC 64.08%, ECE15 17.74%, NLL 0.9186 and Brier 0.4817, stronger than the 0.6B base overall.

Question 7

What happens to inputs that are too long?

Accepted Answer

They are recorded as unsupported_length rather than truncated or silently dropped, and the coverage denominator is documented next to accuracy so a long input does not quietly disappear from the numbers. The default evaluation prints progress every 10 batches and at suite boundaries, and reducing batch size on an out-of-memory error is expected, with the config change recorded.

Jev Dllm

Highlights

Questions about this entry

More Jev resources like this

From the Jev guides

From the Jev community