Question 1

Are the benchmark numbers real?

Accepted Answer

The Jev rows reuse JevBench per-item results on identical ids and the published Nimble holdout figure, while every other row was run here through the same vendored scorer. Full tables, calibration, per-family breakdowns, and methodology ship in the bench report.

Question 2

Should I replace Jev with this in production?

Accepted Answer

Probably not as a drop-in. With a large model behind it, logit readout matches Jev accuracy on the hardest items but not calibration out of the box, and it runs slower and several times more expensively per decision. Small local models fall far short.

Question 3

Will my Jev client code still work?

Accepted Answer

Yes. It mirrors the request and response shape including legend and usage, validation failures, request-id headers, and context budgets, and the official SDK passes unmodified in CI. Key enforcement is opt-in for local use.

Question 4

Can I run it fully offline on a laptop?

Accepted Answer

Yes. A 4-bit mid-size model fits in 8 GB on a laptop GPU and costs about electricity, but expect accuracy far below Jev. Packing answers all questions in one forward pass at a fraction of the cost with some argmax drift.

Question 5

How does it read a decision out of an LLM?

Accepted Answer

No text is ever generated. For each question the model sees the state and lettered options, and the backend reads the next-token distribution at the answer position restricted to the option letters. A noul becomes P(yes), a choice a distribution plus argmax, a score a distribution plus a probability-weighted level. Confidence uses TypeSafe's documented statistic, (n·p_max − 1)/(n − 1).

Question 6

Where is it not wire-compatible?

Accepted Answer

Choice. TypeSafe allows up to 255 options, while jev-sim is capped at 26 because options are read as single-letter codes. Everything else lines up: the 64k per-request and 32k state-plus-longest-question budgets are enforced and return 422 like the real API, request-id headers are returned, and it adds a GET /v1/limits extension the vendor endpoint does not have.

Question 7

Does asking everything in one call change the answers?

Accepted Answer

On the local backend it can. All questions of a request are packed into one sequence so the state is read once, and 8 questions cost about 1.5 times one question on a 4B model, but argmax agreement against separate evaluation is only 74% at k=8. API backends do one call per question instead. Post-hoc temperature scaling on a labelled split is available to recover some of the gap.

Jev Sim

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev community