Docs
OfficialJev Cookbooks
Cookbook: Classification Using Confidence
Classify SEC annual reports into 75 industry groups with one Choice each, then read confidence to decide when to fall back to a broader division.
- Format
- Docs
- Reading time
- 3 min
- Published by
- TypeSafe AI
- Last verified
- Sep 2026
Works forJev for Classification
Used inJev for Software & DevTools
Highlights
- One Choice question with 75 industry-group options classifies each 10-K; a Choice works reliably up to roughly 240 options.
- Across 60 filings a 0.9 confidence cutoff split them in half: the confident half was right 90% of the time, the other half 40%.
- Reporting the unsure half one SIC level up (division instead of group) took that 40% to 70%, lifting useful answers from 39/60 to 48/60.
- Answers return choice, probabilities over all 75 options, and confidence; the recipe reads confidence, not the winner's own probability.
- The SEC file holds 444 four-digit industries that roll into 75 major groups and 10 divisions, assembled with no model calls.
Quickstart
python
CONFIDENT = 0.9
def classify(filing: dict) -> dict:
answer = ask(filing["id"], filing["text"])
sure = answer["confidence"] >= CONFIDENT
return {
"level": "group" if sure else "division",
"label": answer["group"] if sure else division(answer["group"]),
"confidence": answer["confidence"],
"group": answer["group"],
}Watch out
The 0.9 cutoff was tuned on this 60-filing SIC sample, so recalibrate the threshold for other taxonomies or document mixes.
Questions about this entry
Short answers drawn from the linked project or page.
Across 60 SEC filings a 0.9 cutoff split them in half: the confident half was right 90% of the time and the other half 40%. Reporting unsure filings one SIC level up took that 40% to 70%, lifting useful answers from 39/60 to 48/60.
A Choice with 75 industry-group options classifies each 10-K, and the page says a Choice works reliably up to roughly 240 options. The SEC file's 444 four-digit industries roll into 75 groups and 10 divisions with no model calls.
Because confidence tells you how peaked the distribution is, which is what you need to decide whether to fall back. A Choice answer returns the winning option, probabilities over all 75 options and confidence, and this recipe branches on the confidence field, not on how likely the chosen group happens to be.
From sic_codes.tsv, the industry list the SEC publishes for filers to pick their own code from, fetched 2026-08-10: 444 four-digit codes each with an industry title. The digits are the hierarchy, so the first two give the 75 major groups and fixed ranges of them give the 10 divisions, both derived in code with no model involved. 42 of the 75 groups carry no umbrella title in the SEC's list, so each is described by up to eight of the industries inside it.
A label plus how specific it is, at one request per document. It returns the level, group or division, the label at that level, the confidence that drove the choice, and the underlying group so you can see what it would have said at full specificity.
Recalibrate it. The threshold was tuned on this 60-filing SIC sample, so a different taxonomy or a different mix of documents will move where the break sits. The fallback also depends on your labels being hierarchical the way SIC is, since the broad label is derived from the narrow one with no second call.
pip install ipython matplotlib 'cooksafe>=0.2.0,<0.3.0' and set TYPESAFE_API_KEY. Calls cache to json_cache.json so a re-render spends nothing until you delete it. The published figures came from jev-1.12 on 2026-08-12, with the confident threshold set at 0.9 above which the group is reported and below which the division stands in.
More Jev resources like this
A support-ticket cookbook that lets Jev decide which requests are worth remembering, then wraps each call so saved tickets ride along in the state. Six later messages route correctly with memory against two without.
Jev CookbooksDocs#official#cookbook#memory
Official
A live chat filter that asks three questions per message — category, toxicity, and spoilers — then turns the answers into show, highlight, dim, hide, or blur in plain code. The 29-message example hides seven, blurs two, and highlights six.
Jev CookbooksDocs#official#cookbook#moderation
Official
Semantic search over GitHub's Terms of Service: one request ranks all 218 lines with a Choice while a Noul checks whether the document contains an answer at all, including when it should say no.
Jev CookbooksDocs#official#cookbook#search
Official
A two-stage extraction cascade: a mini model extracts, a Noul battery verifies each field in one request, and a 0.7 gate escalates to a reasoning model, sitting on the cost/quality frontier.
Jev CookbooksDocs#official#cookbook#extraction
Official
Decide which of 450 candidate product pairs from two beer catalogues describe the same entity: one three-level Score per pair splits 40 merges, 50 curator reviews, and 360 non-matches.
Jev CookbooksDocs#official#cookbook#entity-matching
Official
Catch wrong and hallucinated citations against RFC 7519 with a string match plus one Choice question: four accurate citations verified at 0.93 or more and all four planted failures caught.
Jev CookbooksDocs#official#cookbook#citations
Official
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.