Jev Sort

Install the CLI globally with npm install -g @cmaintz/jev-sort, or install @cmaintz/jev-sort as a library. Run jev-sort --help for the full flag list. It is published under the MIT license by Christoffer Maintz.

It takes --provider typesafe or --provider cloudflare, with --model jev-latest sent as typesafe/jev on Cloudflare; any other model id is passed through. Set JEV_API_KEY, plus CLOUDFLARE_ACCOUNT_ID when using Cloudflare.

Use --escalate 'conf<0.6' together with --review-out FILE. Rows below the bar go to that JSONL file instead of stdout, and a row counts as confident only if every choice or score answer clears the bar. Exit code is 2 if any rows were flagged, unless you pass --allow-review.

A row whose Jev call fails, or that returns without an answer to every question, is written to --reject-out FILE as JSONL with an _error field. An input line that is not a JSON object gets _line and _raw. The stream keeps going, and the exit code is 3 if anything was rejected.

--thresholds thresholds.json takes the bar from jev-eval instead of a guess. It refuses a file measured on a different set of choice or score questions, warns when the model differs from --model or a gated question was reworded, and prints the accuracy and coverage measured for that gate. Use it instead of --escalate.

On TypeSafe's workflow evals Jev scores 67.8% agreement with frontier-model reference labels, so treat output as pre-labels and spot-check it. Input is text only, and each row must fit Jev's request limit of 32k tokens for state plus the longest question; long documents need summarizing first.

It pays off on thousands of rows or more. For a few hundred rows a one-off LLM script is simpler, and if the rule is a crisp regex or keyword match, plain code beats both. It also gives no rationale for a label and does no counting or date arithmetic, so aggregation belongs downstream.