Do I need a Jev key to try judgekit?

No. The 30-second demo uses only local keyword rules, never loads a provider registry and makes no API call; it prints four JSON decision records and exits 0 when all expected outcomes occur, including a deliberate no-hit. Jev, OpenAI-compatible backends and local NanoJev are optional once you move beyond rules.

What primitives can a judgekit YAML task use?

Classify returns a label, route returns a routing label, score returns a 0-1 score, and verify returns a boolean. Rules run locally and support classify and route; score and verify require a model backend, and native Jev or NanoJev can additionally provide probability distributions in the decision record.

What Python versions and platforms does judgekit support?

Python 3.10 or higher, with CI covering Ubuntu and Windows against Python 3.10 and 3.12. Install from a checkout with `python -m pip install -e .`, and for development `python -m pip install -e ".[dev]"` then `python -m pytest -q`.

What data does judgekit send to a model provider?

run_task passes only the declared input_field, which defaults to text; gold labels and extra metadata stay local. Provider configuration is read from environment variables, and the CLI does not load a .env file automatically, so keys and endpoints must be set first.

What exit codes does the judgekit CLI return in a script?

The judge command exits 1 for an unmatched input, and run --fail-under N exits 2 when the successful-execution rate drops below N. That flag measures the proportion of ok results, not classification accuracy or whether a remote model succeeded.

Does a low-confidence decision fall back to rules automatically?

No. Confidence is gated at 0.7, which caught 3 of 3 misjudgments at 9 percent escalation on 130 items, but the README states low confidence does not automatically trigger a rule fallback; calibration depends on the backend and your data.

How are costs and failed requests recorded?

Each decision records the actual backend, failure reason and configured cost. An unconfigured price, zero subscription marginal cost or zero local spend is not zero total cost, and billed requests that fail response parsing can be undercounted, so check the provider bill.

CommunityJev Tools & CLIs

Judgekit

CLI and Python library that runs declarative YAML judgment tasks - classify, route, score, verify - natively on Jev Choice/Score/Noul, on any OpenAI-compatible backend, or on local keyword rules, with confidence gates, fallbacks and per-decision cost records.

See highlights

Open github.com

Stars: 4
Language: Python
License: MIT
Published by: Community
Last verified: Oct 2026

lexingtonhibiki/judgekit

User repository on GitHub

View on GitHub

YAML-based classification, routing, scoring and verification. Native Jev + OpenAI-compatible backends, rule fallbacks and transparent evaluation. (判官工具箱)

Forks: 0
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

Classify, route, score, or verify with one YAML task. Get a consistent decision JSON and a cost record. Start with free local keyword rules. When you need a model, the same task can use TypeSafe Jev's native API, an OpenAI-compatible endpoint, or local NanoJev. Use it for small, clearly defined decisions in scripts, support workflows, and applications.

Quoted from the project's README. Read the full README.

Tagscommunity yaml classification evaluation

Works forJev for Classification Jev for Routing Jev for Scoring Jev for Verification

Used inJev for Customer Support Jev for Software & DevTools

Languages: Python
Platforms: CLI
Capabilities: Open sourceRuns locally

Highlights

One YAML task can run on Jev, an OpenAI-compatible endpoint, local NanoJev, or rules.
The demo runs offline with local rules and makes no API call: python -m judgekit demo.
Gates low confidence at 0.7, catching 3 of 3 misjudgments at 9 percent escalation (n=130).
Reports 60.0 to 68.3 percent on a frozen 120-item human-labeled spam set at threshold 0.10.
Decision records carry the actual backend, failure reason and configured cost.

Watch out

MIT licensed. Local rules cover classify and route while score and verify need a model; costs use prices you configure, so billed requests that fail parsing can be undercounted. The spam-set threshold was selected in-sample with no holdout.

Questions about this entry

Short answers drawn from the linked project or page.

Related terms

Glossary definitions related to this entry.

Precision and recall

More Jev resources like this

Jev Agent Skill

Claude Code and ZCode skill that offloads classify, route, batch-screen, score and compliance-check judgments to Jev through OpenCode Zen's free tier, bundling a zero-dependency jev.py caller and a Taobao comment-triage pipeline that keeps raw items out of the agent context.

Jev Tools & CLIs#community#claude-code#skills

Community

yuyang2230/jev-agent-skillPython

pyGPT

A desktop AI assistant with a Jev / System One inline plugin for classification, routing, verification, and scoring without chat-style output.

Jev Tools & CLIs#community#python#desktop

Community

szczyglis-dev/py-gptPython

jev-mail-classifier

An IMAP client that turns each inbox category into a Noul, then tags, moves, flags, or calls a webhook when the probability clears that category's threshold.

Jev Tools & CLIs#community#python#email

Community

parth-kp/jev-mail-classifierPython

DocJev

523

Classifies and splits PDFs, DOCX, and PPTX with local page text from LiteParse and category decisions from Jev.

Jev Tools & CLIs#community#python#document

Community

jerryjliu/docjevPython

Gutsy

An Apache-2.0 0.8B decision model that returns calibrated probabilities for yes/no, choice and score questions on CPU. Distributed as GGUF for llama.cpp and served behind a Jev-style /v1/systemone API, so existing Jev clients mostly need only a new base URL.

Jev Models & Reimplementations#community#gguf#local-inference

Community

kouhxp/gutsyHTML

ruling

A local Jev-compatible server that reads Choice, Score and Noul answers from the logits of any MLX checkpoint or OpenAI-compatible endpoint with no training, serving POST /v1/systemone as a drop-in for the official TypeSafe SDK.

Jev Models & Reimplementations#community#mlx#calibration

Community

bradAGI/rulingPython

From the Jev guides

Original write-ups on what this entry is for.

Jev on the Gateways: Vercel, Workers AI, and OpenRouter

Oct 12, 2026

The request shape barely changes across surfaces; the surrounding contract does. When to call the API directly and when a gateway, a Worker binding, or a router entry earns its extra hop.

Jev Tools & CLIs#gateway#vercel#cloudflare

Read article

Exporting Decision Models: GGUF, ONNX, and Device Runtimes

Oct 10, 2026

Taking a decision checkpoint out of Python: what the GGUF and browser WebGPU exports preserve, what they change, and the parity evidence the model cards publish.

Jev Tools & CLIs#open-models#local#onnx

Read article

Records at Rest: Documents, Inboxes, and Risk Queues

Sep 28, 2026

Jev labels a record that already exists. Code moves, flags, or holds it. How the document, mail, and transfer tools keep the action out of the model.

Jev Tools & CLIs#classification#email#security

Read article

Back to all resources

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai