Question 1

Are the benchmark numbers against Jev actually verifiable?

Accepted Answer

No, and that is the main thing to know before reading the tables. Every figure is the author's own measurement, and the column for Jev is a hosted commercial API with undisclosed parameters and no published evaluation harness, so nothing there can be reproduced independently. The macro averages across 95,000-plus samples, the per-domain splits and the Pareto charts are all self-reported. The parts you can check yourself are the properties of the weights: quantization drift, decision retention, and whether the escalation head behaves at the 0.50 threshold once the model is loaded.

Question 2

How is this actually different from calling Jev?

Accepted Answer

Same job, different deployment. Both answer a multiple-choice decision with a probability instead of writing text. The difference the README sells is where the decision happens: Jev is a hosted API, so every call is a network round trip it puts at about 190 ms, while Cerebellum-2B runs locally on a Qwen3.5-2B backbone and picks from a candidate set in one forward pass, with INT8 weights for a 16 GB MacBook and no network egress. You give up a managed service and take on running and quantizing the weights yourself.

Question 3

What happens when the model is not sure?

Accepted Answer

There is a second head for it. ActEscalateHead fuses semantic features with information entropy and escalates at P(Escalate) >= 0.50, handing the case to a human or a System 2 model instead of executing — the README frames this as proactive risk control and bills it as the built-in feature the compared systems lack. It also reports a Brier score of 0.0271 for calibration and a 2.10% rate of confidently wrong picks above 0.8 confidence, both of which you would want to re-measure on your own candidate sets.

Question 4

Does the order of the candidate list bias the answer?

Accepted Answer

That is the specific failure mode the architecture targets. The state side is fully bidirectional with no causal mask, while candidates are isolated from each other by a branch attention mask, which the README reports as 0.60% decision drift when the list is shuffled. For comparison it puts Laya, KEV and Jev in the 88.5% to 91.2% band and GPT-4o at 84.50% with 15.5% position bias. Worth running your own permutation test, since ordering effects are exactly the kind of claim that holds on one candidate set and not the next.

Question 5

How do I run it locally on a Mac?

Accepted Answer

Load the INT8 build with CerebellumModel.from_pretrained and call model.decide with a state and a candidate list; it returns the chosen action and a latency in milliseconds. On a 16 GB M4 MacBook the README reports about 35 to 45 ms per forward pass with MPS, 75 to 95 ms on CPU alone, and roughly 2.23 GB of memory with INT8.

Question 6

What is the 9.1 MB golden rule?

Accepted Answer

The backbone's 186 linear layers are quantized to FP8 or INT8, but the two decision heads are kept at BF16 or FP16 in a heads checkpoint of only 9.1 MB. That keeps temperature scaling and confidence calibration lossless while shrinking the served model.

Question 7

What are the measured quantization trade-offs?

Accepted Answer

Against the full-precision baseline, FP8 drifts at most 0.97% with 100% decision retention at 120.1 ms per read, while INT8 drifts 0.39% at 121.5 ms; both are 2.23 GB, about 36.5% smaller. The README also reports 320-plus requests per second on a single 24 GB card, enough for eight to ten instances.

Cerebellum-2B

Highlights

Questions about this entry

More Jev resources like this

From the Jev guides

From the Jev community