Question 1

The comparison table claims it beats Jev and Laya on most rows. Who ran those numbers?

Accepted Answer

The README says so itself: the Jev, Laya and kev figures come from those projects' own docs and published third-party evaluations, while the Verdict column comes from the bench/ directory in this repo. That is a difference in provenance, not a controlled comparison, and the differences you would act on most — unbounded option count, 100+ languages, no option-order sensitivity — are structural rather than measured.

Question 2

Is the conformal abstain flag real, or just a confidence threshold with better branding?

Accepted Answer

calibrate() fits a temperature and a conformal threshold on held-out data, and the claim is distribution-free: on data like your held-out set the true label lands inside conformal_set at least as often as the coverage you set. That is a stronger guarantee than thresholding a probability, but it is only as good as the held-out split being representative of live traffic, which is your problem and not the library's.

Question 3

Can I point an existing Jev or Laya client at it without rewriting my code?

Accepted Answer

Yes, that is the stated goal of the serve mode: pip install "verdictml[serve]", run verdict serve, and send requests to http://localhost:8000/v1/systemone. The choice, score and noul question shapes are accepted unchanged and answers gain an abstain flag. The native POST /v1/decide endpoint returns the fuller Choice, Score and Check objects if you want the calibration data.

Question 4

Do I need PyTorch installed to run this?

Accepted Answer

Not if you take the ONNX engine: pip install "verdictml[onnx]" pulls int8 weights of about 120 MB and needs only onnxruntime plus tokenizers. The README quotes 0.7 ms per example batched and under 2 ms single on a laptop CPU for that path, and says it is the same weights the browser playground runs, so the numbers do not shift between the two engines.

Question 5

How do I train it on my own labels?

Accepted Answer

Compile a Question with your options, call fit on (text, label) pairs, then calibrate on held-out pairs with a coverage and evaluate; fit trains a head on frozen embeddings, so it takes seconds on a CPU and saves to one JSON file.

Question 6

What is the ONNX engine for?

Accepted Answer

No-PyTorch deployments: verdictml[onnx] uses int8 weights of about 120 MB with onnxruntime and tokenizers only, at 0.7 ms per example batched and under 2 ms single on a laptop CPU, the same weights the browser playground runs.

Question 7

How does it handle long inputs?

Accepted Answer

Inputs longer than the encoder window are chunked with overlap, embedded, mean-pooled and re-normalised, so a twenty-page document becomes one vector instead of a silently truncated one.

Verdict — Manavarya09

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev guides

From the Jev community