Should I trust confidence on its own?

No. Confidence is computed after renormalising and is blind to raw_mass, so an answer can report high confidence on tiny raw_mass; check raw_mass near 1.0 first, and treat low raw_mass as noise rather than a decision.

Does the calibration transfer to real data?

Partly by measurement: a correction fitted on 96 synthetic examples cut ECE substantially on real BoolQ Wikipedia questions, though fitting on BoolQ itself did better; cross-model parameters performed worse than doing nothing.

Will reordering my options change the answer?

For stated facts it did nothing in measurement, but for judgement calls the same option spanned nearly the full probability range and the winner flipped; average over permutations or freeze and calibrate against one exact order.

What actually bounds memory and cost?

Token limits checked before GPU work, with peak residency as state plus the single largest question since the cache rewinds; requests serialize under a lock, the model downloads into the folder cache, and cleanup removes it.

What is it running, and how fast is it?

An off-the-shelf MLX causal LM, Qwen3-1.7B-bf16 in the documented example at about 1.1 GB, which downloads into a .hf_cache folder inside the project directory and is removed by ./cleanup.sh --yes. The document is encoded once and its KV cache kept, so every question is a short rewound suffix: four questions over one document take about 200 ms on an M4 Pro. It is unaffiliated with TypeSafe, though the API shape is deliberately theirs so one client works against either.

How good is it on data it did not invent itself?

On BoolQ, which is real yes/no questions over Wikipedia passages with 3,270 held-out examples and a 62.2% always-yes baseline, Qwen3-1.7B scores 0.754 accuracy raw and 0.780 corrected, at an AUC of 0.852 throughout since calibration only rescales. BoolQ state of the art is around 0.90, so this reads as a laptop-scale result with real signal rather than a competitive entry.

Why does it refuse to apply another model's calibration?

Because a mismatched correction is silently destructive. calibration.json records which model each fit belongs to and the engine refuses a mismatch rather than correcting the wrong thing: the 0.6B's bias of -3.38 applied to the unbiased 1.7B turned a p(yes) of 0.97 into 0.50. The two are miscalibrated differently too, the small one needing a bias shift and the large one a sharpness fix, so one file cannot serve both.

CommunityJev Practices & Patterns

TypeSafe Local

Local experiment that asks an off-the-shelf model typed System One questions with single-position logit readout, measuring speed, calibration, and order sensitivity.

See highlights

Open github.com

Stars: 8
Language: Python
License: MIT
Published by: Community
Author: aabolfazl
Last verified: Oct 2026

aabolfazl/typesafe-local

User repository on GitHub

View on GitHub

Inspired by TypeSafe Ai, Ask a local LLM typed questions, get calibrated probabilities instead of text. Structured output without generation or parsing. MLX / Apple Silicon.

Forks: 0
Last pushed: Updated Sep 2026

typesafe

Repo stats from the GitHub API, cached Oct 2026.

About this project

Ask a local language model typed questions about a document and get back probability distributions instead of text. No parsing, no retries, no API key. TypeSafe shipped a model that returns typed decisions instead of prose, and the idea was interesting enough that I wanted to know how much of it comes from the model and how much from the way it is served. So I built one, on top of an off-the-shelf local model, and measured it.

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import awesomejev

Works forJev for Classification

Used inJev for Software & DevTools

Languages: Python

Highlights

Encodes the document once and reuses its KV cache so each question is a short rewound suffix invisible to the others.
Generates nothing: it applies softmax over legal answer tokens, making wrong-typed answers unrepresentable.
Exposes raw_mass on every answer to separate landed format from noise rescaled into a distribution.
Ships calibration tooling, BoolQ and robustness evals, and token-limit guards checked before GPU work.

Watch out

Raw probabilities mislead until corrected, option order can flip judgement calls at high confidence, and the memory-limit env is only a hint to MLX; size requests with token limits. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-on-a-laptop

Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.