My model says 0.9, why should I distrust that number?

Stated confidence is often miscalibrated, so jeval plots accuracy against confidence with intervals, ECE, Brier and sample sizes to show where high scores actually fail.

How does it turn mistake costs into a threshold I can ship?

It prices every candidate threshold by expected cost per case, draws your current line against the minimum, shades indistinguishable regions and reports the monthly impact of moving.

Does this only work with Jev-hosted classifiers?

Yes. It sits above any probability source including APIs, gateways, local models, regressions and scoring rules. You ingest existing logs or record with its lightweight library.

Why must I avoid the obvious pip install command?

That PyPI name belongs to an unrelated project and this tool is not published there. Use the install script, versioned wheel, or pinned git source instead.

What files does a run produce?

One offline report.html, a thresholds.yaml for your app with the deploy threshold, its uncertainty range and any per-segment lines, a labels.csv of which decisions to label next, and an optional calibration-*.yaml correction map written only when the gain is real.

curl -fsSL the install script piped to sh keeps a private environment in ~/.local/share/jeval and links jeval into ~/.local/bin, with no uv, pipx, root or PyPI; rerun it to upgrade, and install.sh uninstall removes it.

How do I collect the records?

Wrap the client with collect.track(...) where it is built and call collect.resolve(...) where a human settles a case; answers and human answers append under $JEVAL_ROOT/.jeval, and either half works alone since the CLI can ingest a log you already have.

CommunityJev Practices & Patterns

Jeval

Offline evaluator that checks whether classifier confidences match reality and converts mistake costs into a deployable threshold, emitting an HTML report plus machine-readable threshold files.

See highlights

Open github.com

Stars: 22
Language: Python
License: APACHE-2.0
Published by: Community
Author: rlaope
Last verified: Oct 2026

rlaope/jeval

User repository on GitHub

View on GitHub

Measures what your Jev classifier's confidence is really worth, and sets the human hand-off line from what a mistake costs.

Forks: 1
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

What is your classifier's confidence really worth? jeval measures how well a classifier's confidence matches reality, and turns what a mistake costs into the threshold where the machine should stop deciding and a human should start. Install  ·  See it  ·  Get your data in  ·  Read the report  ·  Drift in CI  ·  Commands. Your classifier answers with a label and a confidence. Two questions follow, and jeval answers both:

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import awesomejev

Works forJev for Classification

Used inJev for Software & DevTools

Languages: Python

Highlights

Reliability, cost-curve, impact and segment views built from labeled decision records on disk.
Single offline HTML report plus thresholds file, next-labels list and conditional calibration map.
Library hooks for tracking and resolving cases plus CLI status, report, threshold and drift tools.
Installer keeps a private environment with no extra runtime, server, database or network calls.

Watch out

Findings rest on the labeled records you feed it and thin segments are marked unjudgeable. Threshold advice carries uncertainty ranges and calibration maps are written only on real gain. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-on-a-laptop

Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.