Question 1

Is this a Jev model I can run locally, or something else entirely?

Accepted Answer

Something else. It is an independent project, explicitly not affiliated with TypeSafe or the Jev model, that reimplements the one-pass option-scoring shape: text in, one probability per option out, single forward pass, no token-by-token decoding. It makes no claim about matching TypeSafe's Jev or its private training method, and the only published comparison is against jevlike on identical synthetic data.

Question 2

Is the accuracy win real, given the throughput numbers in the same table?

Accepted Answer

On 800 held-out hard menus with 2 to 8 options, near-miss negatives and distractor sentences, on CPU with matched epochs, batch size and seed, top-1 goes from 0.873 to 0.916 and ECE halves from 0.0367 to 0.0182. The cost is throughput: 40 menus per second against jevlike's 4,608, because of the transformer context encoder. --context-features tunes that tradeoff.

Question 3

Will it generalize past synthetic menus, or only score its own training shape?

Accepted Answer

The published benchmark is entirely hard synthetic menus, and the README says accuracy depends on data quality and split quality as always. It is candid that the hashed n-gram encoder is still shallow next to a pretrained transformer and suggests --encoder hf when meaning matters most. No results on a real-world dataset are documented.

Question 4

Can I reuse my jevlike dataset, and what does a saved checkpoint need?

Accepted Answer

Yes. The JSONL format is the same, so any jevlike dataset trains unchanged, and CSV with context,options,label works too, options pipe-separated, label being the zero-based index with at least two options per row and no duplicates. One catch: a checkpoint stores the trained head and the encoder name, not the frozen weights, so loading an --encoder hf run needs access to the same Hugging Face model.

Question 5

What does jevbetter's evaluation report beyond top-1 accuracy?

Accepted Answer

jevbetter-eval prints top-1, top-3 and top-5 accuracy, MRR, calibration error, accuracy by menu size, throughput, and a shuffled-context control that pairs each menu with the wrong context.

Question 6

Can jevbetter read my data as CSV?

Accepted Answer

Yes, it accepts CSV with context, options and label columns where options are pipe-separated, as well as the JSONL object-per-line format; label is the zero-based index and each row needs at least two distinct options.

Question 7

Does jevbetter need a GPU to train or run?

Accepted Answer

No. The benchmark trains and evaluates on CPU, and the transformer context encoder costs throughput rather than requiring hardware: 40 menus per second against jevlike's 4,608, with --context-features tuning the tradeoff.

Jevbetter

About this project

Highlights

Quickstart

Questions about this entry

Reactions & coverage

From the Jev guides

From the Jev AI model community