Are the benchmark rankings real or just noise and leakage?

No single winner is claimed. At 500 examples the accuracy interval is roughly 2.5 points so several rankings are ties, and SST-2 and AG News were likely seen in LLM pretraining while Banking77 is the cleanest comparison.

How much will a full run cost me in time and API spend?

Use the cheap models for smoke runs and subsets, and the full run only when you need the table. Successful API responses cache in sqlite so re-runs for accuracy are free; use no-cache for fresh latency timings.

What setup catches people before their first real run?

Only OPENROUTER_API_KEY for Jev and the two LLMs; local models need no key and no token. You need uv with Python 3.12, and the first local runs download weights and fine-tune DistilBERT per dataset.

Can I add my own dataset or classifier without forking the runner?

Add a DatasetSpec with repo, split, text column, and label map, or implement the Classifier protocol and register it in build_classifiers. Every classifier picks up the new dataset automatically.

Which classifiers does it compare?

Six: Jev through OpenRouter's Decisions API, the open-weights Laya model, a cheap LLM (gpt-5-mini), a frontier LLM (claude-sonnet-5), a fine-tuned DistilBERT, and zero-shot BART-MNLI.

How do I run a quick sanity check?

uv run jevbench run --smoke --classifiers jev llm-cheap --datasets sst2 runs five examples against the APIs for well under a cent; the full run is about 90 minutes and $9, mostly the frontier LLM.

Why is ECE blank for the LLMs?

Expected calibration error needs per-class probabilities, which only Jev, Laya, and the two BERT variants return; the LLM classifiers produce no probabilities, so their ECE column is empty.

CommunityJev Practices & Patterns

Jevbench — dhruvmehra

Reproducible harness comparing Jev, Laya, two LLMs, and two BERT variants on SST-2, AG News, and Banking77 for accuracy, calibration, latency, throughput, and cost.

See highlights

Open github.com

Stars: 8
Language: Python
License: MIT
Published by: Community
Author: dhruvmehra
Last verified: Oct 2026

dhruvmehra/jevbench

User repository on GitHub

View on GitHub

Benchmark TypeSafe JEV against LLMs, fine-tuned BERT, Laya and zero-shot NLI on text classification: accuracy, calibration, latency, throughput, cost

Forks: 2
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

This is one of several Jev listings for the same project. The canonical entry is JevBench — fstandhartinger.

About this project

Is TypeSafe's JEV a better text classifier than LLMs, BERT, or the open-weights alternatives? This is a small, reproducible benchmark that runs six classifiers over three public datasets and reports accuracy, calibration, latency, throughput and cost side by side. Full tables with macro-F1, ECE, p50/p95 and throughput: docs/results/2026-09-22-n500-summary.md.

Quoted from the project's README. Read the full README.

Tagscommunity awesomejev catalog-import

Works forJev for Classification Jev for Evaluation

Used inJev for Software & DevTools

Languages: Python

Highlights

Tests Jev, Laya, cheap and frontier LLMs, fine-tuned DistilBERT, and zero-shot NLI side by side.
Shares one label-description map across Jev criteria, LLM prompts, and NLI hypotheses.
Measures accuracy, macro-F1, error rate, ECE, p50/p95 latency, throughput, and cost per 1k.
Writes one JSON per classifier-dataset pair plus summary tables with raw provider responses.

Watch out

Fine-tuned DistilBERT sees thousands of labeled examples while the other models see only label descriptions, and latency for API models includes the OpenRouter hop. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-on-a-laptop

Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.