Can I cite the comparison numbers as benchmark evidence?

No. The repo warns that 24 decisions, 12-case flows, and N=5 probes are anecdotes; committed waves in results/ record claimed-versus-measured per claim instead of a leaderboard.

Which of the four providers should I run first?

Start with mock for wiring, then llm or llm-local stand-ins that sample JSON-constrained outputs into the same contract, then real via TYPESAFE_PROVIDER=real with one env flip.

What does the cross-provider compare mode actually measure?

Same fixtures and decide() code across providers: decision agreement, confidence behavior, consistency across reruns, latency and call counts, tokens, and checkout bureau skip rate.

Do I need API keys to learn from this repository?

No key is needed for mock or local paths, and 115 unit tests cover the pure decide() logic and harness; keys are only needed for cloud stand-in or live Jev runs.

What does it take to get a live Jev run working locally?

Node 24, then npm install and npm test for the 115 unit tests on the pure decide() logic. For live calls install the vendor skill with npx skills add typesafe-ai/skills, set TYPESAFE_API_KEY in .env, flip TYPESAFE_PROVIDER=real, and smoke one fixture with node dist/scripts/debug-real.js before any bulk run.

What does a live Jev call actually cost here?

Real Jev emitted 129 to 156 output tokens per ask at output-free pricing, so a triage ask came to about $0.00004 and a checkout ask about $0.00007. The LLM stand-in emitted roughly 100 times more output tokens, concentrating its cost exactly where Jev is free.

Is the speed advantage real or only on paper?

Mixed, and the repo shows both halves. Real Jev measured p50 306 ms on triage and 501 ms on checkout, but p95 of 824 to 1030 ms falls outside the claimed band, while the LLM stand-in ran 30 to 53 times slower per ask and about 220 times slower at p50 wall-clock with five samples and thinking enabled.

CommunityJev Benchmarks & Evaluations

Jev Access Day

Learning scaffold teaching the System One pattern across triage, checkout, dunning, and PR review, with mock, LLM stand-in, and real Jev providers.

See highlights

Open github.com

Stars: 0
Language: TypeScript
License: MIT
Published by: Community
Author: andreaserradev-gbj
Last verified: Oct 2026

andreaserradev-gbj/jev-access-day

User repository on GitHub

View on GitHub

A learning scaffold for TypeSafe AI's System One models: eval harness plus a measured, plain-language comparison of the Jev decision model vs an LLM stand-in on 24 real operational decisions. All numbers reproducible from committed run files.

Forks: 0
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.Homepage

About this project

A learning scaffold for TypeSafe AI's System One models — a new class of AI model that returns typed, calibrated decisions instead of generated text. The first System One model is Jev (early access via waitlist). Live demo: the full measured comparison as an interactive page — jev-access-day.vercel.app.

Quoted from the project's README. Read the full README.

Tagscommunity github

Works forJev for Evaluation

Used inJev for Software & DevTools

Languages: TypeScript

Highlights

One client interface with mock, cloud LLM stand-in, local LLM, and real Jev engines behind one env var.
Four domains share one shape: parallel atomic questions plus confidence-gated decide() code in your code.
Measured waves record calibration, self-consistency, latency, cost, and zero schema violations per claim.
Live interactive demo plus 115 unit tests on the pure routing logic before any paid calls.

Watch out

Explicit learning scaffold, not a benchmark; small samples and anecdotal numbers, unaffiliated with TypeSafe AI. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-evals

Rubric-based eval harness that judges LLM and agent outputs with Jev through the Vercel AI Gateway, answering all rubrics for a case in one round trip.

Jev Benchmarks & Evaluations#community#github

Community