Why add a second model instead of sampling more from the first?

Because grading your own output backfired in measurement. MetaCog requires a different judge model, with hosted Jev as default, local Reflex, or a keyless logit judge as alternatives.

Are the accuracy gains real once time and token cost count?

Paired runs showed gains with significance reported, but extra paths cost thinker tokens and wall time. Triage, sketch outlines, and best-of-N trade accuracy against latency instead of using the full adaptive default.

What if I have no API key for a judge?

Use LogitJudge with a local gguf or a server returning top logprobs; it reads P(yes) in one forward pass without generating text. It tied Jev closely on saved pools but runs slowly on CPU, so use a GPU.

What does my thinker server need to support?

Any OpenAI-compatible server works for best-of-N. Stepwise needs prefix continuation, race needs streaming plus parallel serving, otherwise fall back to prompt mode or best-of-N.

What are the four modes and when would I pick each?

adaptive is the measured default. best_of_n samples full answers and is the cheapest, working with any chat endpoint; stepwise judges short continuations and needs prefix continuation; race streams all paths at once and cancels losers, giving lower latency but needing server parallelism.

What packages do I install, and what does each extra add?

pip install -e . gives the core with httpx and pydantic only. The [local] extra adds a llama-cpp logprob judge, [eval] adds the HumanEval harness, and [hf] adds an in-process transformers thinker.

Can I plug in my own thinker or judge?

Yes. Any object with a generate(...) method works as a thinker, so an OpenAI-compatible server, a Hugging Face checkpoint or your own sampler all fit. A judge is any object with score(problem, candidates) -> Verdict, and the one rule is that it must not be the thinker.

CommunityJev Agents & MCP

MetaCog

Python inference-time loop that lets a separate judge score one greedy answer and, when uncertain, samples extra thought paths in parallel and returns the best-scored path.

See highlights

Open github.com

Stars: 25
Language: Python
License: MIT
Published by: Community
Author: ItIsCuthNotCup
Last verified: Oct 2026

ItIsCuthNotCup/MetaCog

User repository on GitHub

View on GitHub

Metacognition for any agent. Drastically improves accuracy with almost no increase in cost.

Forks: 2
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

Let a small, fast judge steer a bigger model's reasoning. MetaCog wraps any language model — local or API — in an inference-time metacognition loop. The model writes one answer; a separate judge scores it. If the judge is confident, that is the answer. If not, the model writes several more thought paths in parallel, the judge scores each path and each distinct final answer, and the best one is returned. No training, no weights touched.

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import

Works forJev for Classification

Used inJev for Software & DevTools

Languages: Python

Highlights

Adaptive default stops at high confidence, otherwise sizes extra branches from uncertainty.
Scores every full path plus each distinct bare final answer with an answer prior before picking.
Modes for best-of-N, stepwise continuation, and streaming race with early cancellation of losers.
Jev hosted by default, Reflex local, or keyless LogitJudge from next-token logits; judge must differ from thinker.

Watch out

The judge must not be the thinker; self-grading measured worse than no judge. Race mode needs parallel serving or timing collapses, and model-generated code should run containerized. Report coverage ceiling next to accuracy. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-recruiter

Local LinkedIn recruiting workspace on Browser Use's Jev Ultrafast: Jev screens titles, drives navigation, and assesses profile excerpts against your brief while a live browser stays visible; matches and evidence export as JSON.