Janus

That is the tool's whole argument: it ships no default threshold, it measures one. Start with --sample 20 --seed 1 to check the wiring and the real cost per call, set --budget to stop a run that would overspend, and an interrupted run resumes by id without re-paying for a completed call.

On Banking77 it did: 80.2% at threshold 0.67, +1.4% over the better single model alone, at $0.1033 and a 302ms median against $0.2207 and 2269ms. On Web of Science no threshold beat the better single model, and measure returned DO NOT ROUTE, telling you to run that model by itself.

Yes, through --log, but then the report is about how often the logged model agrees with a reference model, not how often it is right. There is no ground truth in a log and the tool refuses to print a word that would suggest one. With --dataset and a gold label the report is about accuracy.

That is why you pin the resolved version when routing, resolve to something like typesafe:jev-1.13.0 rather than the alias, since an alias moves when a release ships and a threshold measured on one version does not transfer. Aliases are fine in measure, which records what the API answered. janus check reports model drift and Router raises rather than applying a stale threshold.

On Banking77 at 0.67 the router auto-handles 88.4 percent of traffic at 80.2 percent accuracy, 1.4 points above the better single model alone, for $0.1033 and a 302ms median decision against $0.2207 and 2269ms for the fallback alone. That is 53 percent less money, an 87 percent lower median latency, and 11.6 percent escalation. The report also prints a ceiling of 83.2 percent for that pair of models, which is the task's limit rather than the routing's.

A labelled JSONL with one object per line carrying id, text and gold_label, plus a label file naming every class you allow, with an instructions line and a label-to-meaning map. The tool's own runs used 500 examples per dataset, and the README says start with a few hundred labelled rows because larger samples generally give more stable estimates. Banking77 is distributed under CC BY 4.0, with provenance and citation in the repository's data README.

An agent skill ships in the repository, installable with npx skills add FirasSX914/Janus --skill janus-decide, or as a Claude Code plugin via marketplace add FirasSX914/Janus then plugin install janus@janus, invoked as /janus:janus-decide. The skill describes the commands of the version it ships with and adds nothing; it will not invent a log, a label, a question or a number, and says which one is missing and stops.