Question 1

Which classifiers can it compare?

Accepted Answer

TypeSafe Jev (jev-latest), OpenAI Decisions, Cloudflare Clef and Clef Flash, and any Claude model such as Haiku, Sonnet or Opus, all driven through one standard-library HTTP stack.

Question 2

Can I add my own tasks?

Accepted Answer

Yes. Any directory of task files in the same JSON shape works with --corpus-dir, and the corpus can be narrowed by suite, task, domain, sample fraction or per-task cap.

Question 3

How do I run it, and do I need keys?

Accepted Answer

pip install classifier-bench, then classifier-bench tasks to see the corpus, classifier-bench estimate for a registry-priced cost with no calls, and classifier-bench run --dry-run for synthetic providers with no keys and no spend. A real run writes to runs/<name> and needs the relevant credentials: JEV_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY, or CLOUDFLARE_AUTH_TOKEN with CLOUDFLARE_ACCOUNT_ID. classifier-bench analyze then produces metrics.json and report.md offline.

Question 4

What is actually in the corpus?

Accepted Answer

Two corpora that are never pooled, each getting its own headline. The authored suite is 16 tasks, 434 cases and 539 decisions written as decision rules so every label follows from them, with a blind second annotator agreeing on 431 of 434. The committed public suite is 10 tasks and 1,261 cases of seeded, label-stratified samples from BANKING77, CLINC150, MASSIVE, Civil Comments, DKHate, SMS Spam, PAWS and GoEmotions. Five more tasks are download-only and pulled through the Hugging Face rows API.

Question 5

How does it keep the comparison fair?

Accepted Answer

Every classifier gets the same instructions and criteria, and no prompt is tuned per model. One standard-library HTTP stack handles all of them, with zero retries, a 10-second default deadline, no prompt caching and wall-clock timing around the single call. Each repeat is a seeded shuffle and the classifier going first rotates from case to case; warmup pairs run first and are excluded. The run directory freezes protocol.json, cases.json and registry.json before the first call, appends attempts.jsonl, and seals everything with SHA256SUMS.

Question 6

What does the report measure?

Accepted Answer

Match with the expected labels including fallback answers, macro F1, recall and latency per label, and top mismatches. Then latency percentiles from p25 to p99, reliability counters for 200s, 429s, 529s, timeouts and fallback decisions, token spend priced off the registry, calibration as ECE, MCE and Brier with a reliability table, self-consistency across repeats, head-to-head ratios with clustered-bootstrap intervals plus Cohen's kappa and an exact McNemar test, an escalation cascade sweep from t=0.5 to 0.99, context scaling padded up to 32k, and load runs at 1, 4 and 16 concurrency.

Question 7

What does a full pass cost, and can it stop me overspending?

Accepted Answer

Rough registry prices for one pass over all 2,321 cases: Jev $0.02, Clef Flash $0.05, Decisions $0.07, Clef $0.14, Haiku $2.30 and Sonnet $4.65. run prints the estimate before spending anything and refuses when it exceeds --max-cost, which defaults to $5. Narrow a run first with --suite, --tasks, --domain, --sample, --max-cases and --repeats, and note that small samples give wide intervals.

Classifier Bench

About this project

Highlights

Questions about this entry

Related terms

More Jev resources like this

From the Jev community