Jev Benchmarks

The pilot is anchored tightly: pinned dataset revision fef2a2ac62b69c58670047dddf045c53d7c3cb5e, pinned GLiNER checkpoint 235cf92d6d4318da9bfca0d08975c8fa7250d13b, seed 20260917, a protocol tag created before inference, resolved model jev-1.13.0 recorded from every response, SHA-256 hashes of config, manifest and prediction logs, and 2,000 bootstrap resamples. Contamination is listed as a limitation since public data may be in either model's training set.

On DAIR Emotion the accuracy gap is unresolved at 0.480 versus 0.440 with an interval of [-0.070, +0.150] straddling zero, and calibration is substantially worse: Brier 0.846 against 0.668, NLL 5.588 against 1.381, and zero probability on the true label for 16% of examples. Coverage at the 5% error budget is 0.000 for Jev against 0.020 for GLiNER.

No. The README says those thresholds are selected and evaluated on the same pilot slice, so the coverage figures are descriptive rather than something to ship. It also notes most Banking77/BTZSC classes have one or two sampled examples, making per-class estimates unstable, and that the 72-hypothesis, 200-row configuration with no positive candidate is excluded rather than assigned a fabricated class.

Not as a normalized model comparison, and the README says so. GLiNER2.5 ran locally on an Apple M4 Max CPU while Jev was called as a hosted service from France. GLiNER won the 4- and 6-label tasks at roughly 44 ms p50 against 236-256 ms, while Jev was slightly faster on the 72-label condition at 246 ms against 296 ms p50. These are operational measurements of two different deployments.

Python 3.11 to 3.13 and uv, then `uv sync --extra benchmark --dev`. `jev-bench prepare` builds the evaluation manifest without calling either model, and `jev-bench run --config configs/pilot-v1.yaml --backend gliner` then `--backend jev` writes append-only JSONL that skips successful examples on resume. `jev-bench report` builds the JSON and Markdown reports. The ML runtimes stay optional behind the data, gliner, jev, and benchmark extras.

No. Raw examples and predictions live under the ignored results/runs/, while the aggregate results in results/reports/ contain no example text and no credentials. The Jev SDK also honors TYPESAFE_BASE_URL and TYPESAFE_DEFAULT_MODEL, so you can point at a different endpoint or model. Re-running prepare refuses to overwrite a different manifest in the same experiment directory, and the README's rule is never to commit credentials.

Yes. A backend implements warmup, predict, and close and returns the common Prediction record, and a dataset adapter produces Example records with an ordered label tuple and a single target index. A new comparison is expected to pin dataset and model revisions, compare only capabilities with equivalent output contracts, add a fixture-level contract test needing no network or accelerator, and preserve raw probabilities and failures before summarizing.