How much did the discovery loop improve?

Held-out RMSE fell from 3.09 predicting the mean to 1.77 after five rounds, stopping at 2.47 for word count, 2.15 for the TypeSafe score, and 1.87 for one proposal along the way, across 800 unseen reviews.

What is the scaling limit?

Request count grows with rows, not questions: 100,000 rows means 100,000 requests per round. The example runs one request per row per round with 8 concurrent workers.

What data is it running on?

2,000 WineMag reviews, split into 1,200 development rows that drive the loop and 800 held-out rows scored once at the end. Each row is a tasting note in text and the critic's score on an 80 to 100 scale out, and RMSE is measured in critic-score points where larger misses count for more.

How do 38 questions turn into 67 numeric columns?

Each feature is proposed as an intensity or a presence. Intensity features become Score questions, and each score answer contributes two columns, the average level it points at and how spread out it is around that average, so 29 score questions give 58 columns. Presence features become Noul questions, one probability each, so 9 give 9 columns. CatBoost trains on the 67.

How does the loop actually work?

An LLM proposes the questions, TypeSafe answers them for every row, and CatBoost trains on the answers. Then CatBoost reports which questions it used and which rows it still gets wrong, the next proposal call reads that report, and the loop runs again. Most of the gain lands in the first proposal call, which reaches 1.87 on its own, and four further rounds of reading its own worst predictions take it to 1.77.

What rubrics does it ask with?

Intensity uses five levels from not present at all through dominant, presence uses Noul criteria where true means the note states it or clearly implies it and false means there is no indication. There is also a direct ask, ten quality bands from faulty or unpleasant to profound, rescaled onto the 80 to 100 critic scale, which is the 2.15 baseline.

How do I run it, and how is the model configured?

The notebook needs CatBoost alongside cooksafe and the standard plotting stack, with TYPESAFE_API_KEY set. It uses jev-1.12, 5 folds and 3 repeats to steady the error at this sample size, and a CatBoost regressor of 400 iterations at depth 4, learning rate 0.05, RMSE loss and a fixed seed. Calls are cached to json_cache.json so a re-render spends nothing; delete it to run live.

Docs

OfficialJev Cookbooks

Cookbook: Autoresearch Feature Discovery

Turn wine tasting notes into 67 numeric columns for a CatBoost regressor: five rounds of reading its own errors cut held-out RMSE from 3.09 to 1.77 across 800 unseen reviews.

See highlights

Open docs.typesafe.ai

Format: Docs
Reading time: 3 min
Published by: TypeSafe AI
Last verified: Sep 2026

Tagsofficial cookbook research extraction confidence

Works forJev for Scoring Jev for Extraction

Used inJev for Software & DevTools

Highlights

2,000 WineMag reviews: 1,200 dev rows drive the loop and 800 stay held out, scored once at the end.
38 questions became 29 Score questions (mean plus spread, 58 columns) and 9 Nouls, for 67 features total.
RMSE fell from 3.09 predicting the mean to 2.47 word-count, 2.15 TypeSafe score, 1.87 one proposal, and 1.77 after five rounds.
One request per row per round, run with 8 concurrent workers.

Quickstart

python

def feature_questions(features: list[dict]) -> dict:
    questions = {}
    for feature in features:  # intensity -> Score, presence -> Noul
        if feature["kind"] == "intensity":
            questions[feature["name"]] = Score(
                instructions=feature["question"], criteria=INTENSITY_LEVELS)
        else:
            questions[feature["name"]] = Noul(
                instructions=feature["question"], criteria=PRESENCE_CRITERIA)
    return questions

response = client.system_one(  # 38 questions x 2,000 notes -> 67 model columns
    state=note, questions=feature_questions(features), model=TYPESAFE_MODEL)

Watch out

Request count grows with rows rather than questions: 100,000 rows means 100,000 requests per round.

Questions about this entry

Short answers drawn from the linked project or page.

Reactions & coverage

Posts, threads, and videos about this entry from around the web.

X: A 26-sheet plan set in 2.9 seconds

X: Which exam questions come back

X: 900 images in 40 seconds

From the Jev guides

Original write-ups on what this entry is for.

Retrieval with Decisions: Filter, Rerank, and Verify

Oct 12, 2026

Retrieval is a sequence of bounded judgments. How a decision model slots into passage filtering, semantic find, reranking, and citation checks, and what stays in deterministic code.

Jev Cookbooks#retrieval#reranking#rag

Read article

One Request, Many Questions: Batching, Fan-Out, and Composite Scoring

Sep 12, 2026

Jev bills input tokens and gives output away, and questions in one request run in parallel. The design consequence is simple: ask more per call, then decide in code.

Jev Cookbooks#batching#patterns#search