Why put Jev in front of an agent at all?

Classic filters are another slow LLM call or plain regex, so teams skip filtering. At hundreds of milliseconds per check versus seconds for an agent call, the filter stops being a trade-off and avoided agent cost is counted.

Does the LLM judge still matter?

Yes, but only where it adds something unique: explanation. Confident rejections escalate for the judge to explain, borderline approvals escalate for a second opinion, and clean approvals end there.

How do I know the policy thresholds are sane?

Move a threshold in the live tuner and watch queued cards migrate between piles with no new calls spent, then check the calibration study for agreement, calibration error, and cost curves before trusting a cutoff.

What does the backend actually run?

A LangChain agent with order-lookup and ticket-opening tools under a business policy of refunds, exchanges, and cancellation windows, reached via the evaluate endpoint with chat reserved for agent and judge.

What criteria does the Jev Lab guardrail check?

Seven typed criteria in parallel: prompt injection, out of scope, sensitive data, abusive language, human request, intent for routing and urgency, feeding five actions PASS, ROUTE, ASK, HANDOFF and BLOCK.

How does Jev Lab decide when to escalate to the LLM judge?

Two reasons only: reprovado, where Jev is certain there is a problem and the judge is there to explain rather than confirm, and incerto, where Jev approved by a thin margin and a second opinion is worth it.

How does Jev Lab reach Jev?

Jev is not a chat-completions model on the Vercel AI Gateway; it uses its own POST /v1/evaluate endpoint with a model, state and questions body, and lib/to-jev-state.ts translates the app's messages and enforces the Gateway's 32k token Jev limit.

CommunityJev Benchmarks & Evaluations

Jev Lab — JoaoClemer

Portuguese-language customer-service study with two Jev experiments: a pre-agent guardrail filtering inbound messages and a per-turn pre-evaluator escalating only failures and borderline calls to an LLM judge.

See highlights

Open github.com

Stars: 1
Language: TypeScript
License: MIT
Published by: Community
Author: JoaoClemer
Last verified: Oct 2026

JoaoClemer/jev-lab

User repository on GitHub

View on GitHub

O Jev (TypeSafe AI) como guardrail em tempo real e como pré-avaliador de agentes conversacionais de negócio. Next.js + LangChain via Vercel AI Gateway, com estudo de calibração em português.

Forks: 0
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

This is one of several Jev listings for the same project. The canonical entry is Jev Lab — jammaru.

About this project

Um estudo prático do Jev) aplicado a agentes conversacionais de negócio — o modelo da TypeSafe AI que não escreve texto, só toma decisões. O Jev foi lançado em setembro de 2026 e virou assunto rápido: um modelo da classe que a TypeSafe chama de System One, que não gera texto — recebe um estado e perguntas tipadas, e devolve decisões com probabilidade calibrada. A promessa divulgada é de ser dezenas a centenas de vezes mais rápido e mais barato que um LLM nas tarefas de classificação e julgamento.

Quoted from the project's README. Read the full README.

Tagscommunity github

Works forJev for Tool calling

Used inJev for Software & DevTools

Languages: TypeScript

Highlights

Guardrail checks seven criteria in one parallel pass and maps them to PASS, ROUTE, ASK, HANDOFF, or BLOCK.
Live threshold tuner reprocesses the queue in the browser from cached Jev answers with no new calls.
Eval side grades six criteria per agent turn and sends only rejections and near-misses to the slower judge.
Calibration pipeline measures human agreement, calibration error, repeat variance, and cost-versus-agreement curves.

Watch out

Demo workload costs and latencies were measured on the author's gateway plan and credit; the app renders without a key but calls return explicit errors. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-evals

Rubric-based eval harness that judges LLM and agent outputs with Jev through the Vercel AI Gateway, answering all rubrics for a case in one round trip.

Jev Benchmarks & Evaluations#community#github

Community