Is 21 out of 21 a real evaluation?

It is a smoke test, and the author says so. The 21 hand-written scenarios were written by the same person who wrote the question, and the current wording was picked from three candidates on a separate 16-scenario dev set that is not even in the repository.

What happens if Jev is wrong, slow, or unreachable?

Every failure path wakes the agent. A throw, a call over five seconds, or a missing key returns reason error with the cause attached, and a probability between 0.2 and 0.5 returns unsure and wakes. Only under 0.2 skips, so the failure mode is a wasted turn rather than a dropped event.

Could a hostile email keep my agent asleep?

It could try, since the event and observation are untrusted input and the README admits a crafted email might argue itself into the unrelated option. The question tells Jev they are data and maxSkips bounds the damage at ten skips in a row, but the author is clear this is not a security boundary.

Is a 250 ms judgment call cheaper than the agent turn it avoids?

Unproven. Calls measured 253 ms at p50 and 519 ms p95 over 21 runs, and the README states outright that savings are not measured and there are no production numbers. Whether it pays off depends entirely on how often your agent would otherwise wake for nothing.

How do I install and configure it?

Install from GitHub with npm install github:shitianfang/wakegate ai, where ai (the Vercel AI SDK, 7.0.105 or later) is a peer dependency. The default model is typesafe-ai/jev on the Vercel AI Gateway, which reads AI_GATEWAY_API_KEY.

Which paths wake the agent without asking Jev?

A user message always wakes as user-message, a bare timer with no event or observation wakes as nothing-to-judge, skipped at or above maxSkips or a missing skip count wakes as skip-limit, and an error, timeout over five seconds or missing key wakes as error.

What exactly is sent to Jev on a call?

Only waitingFor, event and observation, sent to Jev through the Vercel AI Gateway or whichever model you pass. Nothing else leaves your machine, and wakegate fetches nothing on its own, so a vague waitingFor or a stale observation gets a poor verdict.

Jev directory

CommunityJev Tools & CLIs

Wakegate

An experimental TypeScript gate that asks Jev one wake, not-yet, or unrelated Choice before resuming a sleeping agent's LLM, while code handles timers, user messages, skip limits, errors, and timeouts.

See highlights Open github.com

Stars: 6
Language: TypeScript
License: MIT
Reading time: 3 min
Published by: Community
Author: shitianfang
Last verified: Sep 2026

shitianfang/wakegate

User repository on GitHub

View on GitHub

Ask Jev whether a sleeping agent's wakeup is worth a full LLM turn before you resume it. A fail-open wake gate for long-running agents on Workers, Durable Objects and Node.

Forks: 1
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

Before you resume a sleeping agent, ask Jev whether the wakeup is worth a full LLM turn. Long-running agents sleep. A timer fires every 30 minutes to re-check a price, or an email lands while the agent waits for one particular reply. Each wakeup usually resumes the LLM with its whole context, and many of them end in "nothing changed, back to sleep". That turn's tokens are spent anyway.

Quoted from the project's README. Read the full README.

Tagscommunity typescript agents vercel cloudflare

Works forJev for Evaluation

Used inJev for Software & DevTools

Highlights

Jev only gates the resume: the wakeup is skipped only when Jev puts less than 0.2 on wake, and every other path wakes the agent.
User messages, bare timers, skipped >= maxSkips, Jev errors, 5-second timeouts, and a missing key all wake without asking Jev.
In a local workerd Durable Object alarm an unchanged $349.99 kept the agent asleep (p = 0.04) and $279.99 against a below-$300 goal woke it (p = 0.99).
A Jev call took about 250 ms in their runs; the whole gate is src/index.ts at 69 lines.
Its eval is 21 hand-written scenarios, 11 that should wake and 10 that should not, not a benchmark.

Quickstart

import { shouldWake } from "wakegate";

const verdict = await shouldWake({
  waitingFor: "A reply from Dana Li about scheduling my second-round interview",
  event: { type: "email", from: "[email protected]", subject: "Your Daily Digest" },
  skipped: 0,
});

if (verdict.wake) await resumeAgent();

Watch out

MIT-licensed. Not on npm yet, so it installs from GitHub; needs the Vercel AI SDK 7.0.105 or later as a peer dependency, the 0.2 threshold and 5-second timeout are fixed, and savings are not measured.

Questions about this entry

Short answers drawn from the linked project or page.

Reactions & coverage

Posts, threads, and videos about this entry from around the web.

X: Search and tagging on keep.md

X: Jev lands on the Vercel AI Gateway

X: Vercel ships the AI SDK provider for Jev

Reddit: Jev is on Workers AI as typesafe/jev

From the Jev guides

Original write-ups on what this entry is for.

Jev on the Gateways: Vercel, Workers AI, and OpenRouter

Oct 12, 2026

The request shape barely changes across surfaces; the surrounding contract does. When to call the API directly and when a gateway, a Worker binding, or a router entry earns its extra hop.

Jev Tools & CLIs#gateway#vercel#cloudflare

Read article

Jev at home: voice pipelines, Assist routing, and sentinels

Oct 16, 2026

How Home Assistant conversation agents, macOS voice pipelines, and an advisory sentinel put a fast typed decision in the loop — one call per utterance, thresholds in code, and a daily token budget.

Jev Tools & CLIs#home-assistant#voice#automation

Read article

Search visibility, audited by decisions

Oct 16, 2026

Jev-based SEO tools split the crawl from the call: deterministic crawlers count and parse, while typed judgments rank fixes, place internal links, score search results, and track AI-answer mentions — with costs and caveats attached.

Jev Tools & CLIs#seo#search-visibility#audits

Read article

From the Jev AI model community

Posts from builders shipping with Jev right now.

Follow @typesafeai

Wakegate

About this project

Highlights

Quickstart

Questions about this entry

Is 21 out of 21 a real evaluation?

What happens if Jev is wrong, slow, or unreachable?

Could a hostile email keep my agent asleep?

Is a 250 ms judgment call cheaper than the agent turn it avoids?

How do I install and configure it?

Which paths wake the agent without asking Jev?

What exactly is sent to Jev on a call?

Reactions & coverage

More Jev resources like this

From the Jev guides

From the Jev AI model community