CommunityJev Practices & Patterns
Auto-optimizing Jev: Half the Errors, 1/7 the Cost
Kiln points its autoresearch loop at Jev for Mandarin pinyin disambiguation. The optimized harness, built from jieba segmentation and CC-CEDICT word evidence with some chained Jev calls, cuts mistakes from 316 to 147 on 61,750 decision positions, ahead of a DeepSeek reasoning baseline at $1.60 versus $11 per pass.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Works forJev for Classification
Used inJev for Education
Highlights
- Optimized harness: 147 mistakes versus 285 for DeepSeek V4.1 Flash with reasoning on, at $1.60 versus $11 per pass and under 300 ms versus about 8,500 ms.
- Unoptimized Jev made 316 mistakes at $1.22; the pypinyin phrase table made 1,434; the whole campaign cost about $18.
- Word evidence (jieba plus CC-CEDICT) generalized from 12 worked characters to 20 unseen ones: 97 mistakes down to 11, none made worse.
- Chained calls settled readings for some characters (the character yao: 28 mistakes to 0) and were kept only where they measurably helped.
Watch out
One campaign by Kiln AI on a single pinyin task with a 1,085-text corpus, and the post treats differences under about fifteen positions as run-to-run noise. Jev receives no fine-tuning and no step-by-step reasoning.
Questions about this entry
Short answers drawn from the linked project or page.
It built word evidence per sentence: segment with jieba, look up every CC-CEDICT word covering the marked character, report the reading each word gives and whether the segmenter keeps the word together, then let Jev choose the reading.
On Kiln's corpus, yes: 147 mistakes versus 285 for DeepSeek V4.1 Flash with reasoning on across 61,750 decision positions, at about a seventh of the cost and a thirtieth of the latency.
No. The post describes Jev as a classifier that generates no text, so there is no chain of thought to read, nothing to fine-tune and no way to ask it to think step by step. The autoresearch loop changes only the state, instructions and criteria presented to it.
It scores 61,750 decision positions drawn from 1,085 human-edited texts, where a character has more than one reading. Across the wider corpus 184,372 characters were scored, and 33.5% of them needed a decision at all.
It made 147 mistakes against 285 for DeepSeek V4.1 Flash with reasoning on across 61,750 decision positions, at $1.60 versus $11 per pass and under 300 ms versus about 8,500 ms.
Unoptimized Jev made 316 mistakes at $1.22, and the pypinyin phrase table made 1,434. The whole autoresearch campaign cost about $18.
Yes. Word evidence built from jieba segmentation plus CC-CEDICT lookups generalized from 12 worked characters to 20 unseen ones, cutting 97 mistakes to 11 with none made worse. Chained calls settled some characters, such as yao from 28 mistakes to 0, and were kept only where they measurably helped.
More Jev resources like this
87GitHub stars
WindTunnel is a benchmark harness for browser agents. It runs the same tasks on the same self-hosted Docker sites across every interaction mode, WebMCP, code execution, DOM, accessibility tree and screenshots, and scores the resulting application state instead of judging the run.
Jev Practices & Patterns#community#jevbest
Community
nekuda-ai/WindTunnelHTML
nekuda-ai/WindTunnelHTML62GitHub stars
Self-hosted workbench that screens JSONL or CSV into keep, review and reject partitions, then fine-tunes your own model and reports loss and perplexity on a held-out split.
Jev Practices & Patterns#community#awesomejev
Community
RenaGao/jev-dataopsPython
RenaGao/jev-dataopsPython38GitHub stars
Local unofficial /v1/systemone adapter that serves Choice, Score, and Noul probabilities via first-token LLM judgments or a fine-tuned ModernBERT scorer, including Japanese docs.
Jev Practices & Patterns#community#awesomejev#catalog-import
Community
Argos1111/jev_localPython
Argos1111/jev_localPython32GitHub stars
CLI that calibrates Jev questions against your own labels: write questions, label examples in JSONL, run check, and get a verdict per question — gate, gate-above-confidence, ranker, unusable, or too-few-examples.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
smkrv/jev-calibrateTypeScript
smkrv/jev-calibrateTypeScript24GitHub stars
Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
rorshopping/jev-on-a-laptopPython
rorshopping/jev-on-a-laptopPython23GitHub stars
Live race rig that labels 1,000 real app reviews with Jev typed questions against Gemini JSON output on identical batches, then replays recordings and re-ranks rows locally without extra model calls.
Jev Practices & Patterns#community#jevbest#catalog-import
From the Jev guides
Original write-ups on what this entry is for.
Most Jev accuracy problems are state problems. What belongs in a state, what is a pre-made judgment to leave out, and why naming the slice you ask about matters more than the wording.
Jev Practices & Patterns#state#design#primitives
Read article
A Noul returns one number and no separate confidence. How to read the value, pin its boundary with criteria, split compound questions, and set a per-action threshold that matches the cost of being wrong.
Jev Practices & Patterns#noul#primitives#confidence
Read article
A Score is a weighted position, not a ruler. Why the number can land between levels, why levels are judged separately, and how to combine several atomic scores in code instead of one crowded rubric.
Jev Practices & Patterns#score#primitives#scoring
Read article
From the Jev community
Posts from builders shipping with Jev right now.