The published report compares two local models. Was any of it measured against Jev?

No, and the README says so up front. The launch run happened on a machine that could not hold the larger local model, so the star was a 0.8B model served locally in bfloat16 and the double was a 421M model through the bundled sidecar, with no TypeSafe key on the box. The verdict on every suite was do not route at a five percent error budget. The configuration example is what you would use to rerun it with Jev as the star.

If the local model agrees with Jev, does that mean the local model is right?

No, and the README puts this first in its limits: agreement is measured against the star, not against truth, so a double that agrees with a wrong star is wrong too. That is what the labeled suites are for, where accuracy, Brier score and coverage at a five percent error budget are reported for both sides so you can see who is actually right. Without labels you have agreement and latency, nothing about correctness.

Does putting the proxy in the path slow down or change what my app gets back?

Neither, by design. The star still answers every request, the proxy forwards the body unchanged and returns the reply byte-for-byte, and the doubles never delay or alter it. If a double times out or errors, the record says so and the report counts it as unavailable rather than failing the run. The only way your app can tell the proxy is there is two extra response headers. Each request writes one JSON line per day with the request, its hash and every backend's timing.

Can I evaluate a new model checkpoint without paying for Jev again?

Yes, that is what replay is for. Once traffic has been recorded against the paid star, you can rerun the same requests against any new checkpoint, and the star's recorded answers are kept and marked as recorded so only the new double is called. That makes evaluating an open model against real recorded traffic cheap. Without live traffic you can import a labeled suite instead, which is how the accuracy and calibration numbers get produced at all.

How do I set up the proxy?

Run npx stuntdouble serve --config stuntdouble.json, with a star block naming Jev's URL, model and per-million price and a doubles array, then change one line in your app, the SDK base URL, to point at the proxy.

What does the report tell me?

Label agreement per question and primitive, policy agreement wherever the suite carries policy code, confidence-gated agreement, a swap verdict with the threshold and saving per 1,000 decisions, accuracy, Brier, ECE and coverage on labeled sets, and p50 and p95 latency, availability and cost.

What does it record, and can I avoid storing state?

One JSON line per request in records/ .jsonl holding the request, its SHA-256, the optional label and every backend's status, model, answers, usage and elapsed milliseconds; pass --no-state to store only the hash.

CommunityJev Practices & Patterns

Stuntdouble

A zero-dependency Node proxy that sits in front of Jev, forwards every request unchanged, and reports whether a local decision model would have made the same call on your own traffic.

See highlights

Open github.com

Stars: 1
Language: JavaScript
License: MIT
Published by: Community
Author: ReallyArtificial
Last verified: Oct 2026

ReallyArtificial/stuntdouble

Organization repository on GitHub

View on GitHub

Drop-in /v1/systemone proxy that shadows Jev with local decision models (Kev, Laya) and reports whether you can swap

Forks: 0
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

stuntdouble is a zero-dependency Node proxy that sits between your app and TypeSafe's Jev API: you change one line (baseURL) and your app keeps getting Jev's exact answers, while the proxy quietly sends every request to one or more local decision models (Kev, Laya, anything on the same wire shape), records all the answers, and then produces a report that tells you whether the local model would have made the same decision, at what confidence, how much faster, how much cheaper, and, uniquely, whether your…

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import

Works forJev for Classification

Used inJev for Software & DevTools

Languages: JavaScript

Highlights

Change one base URL in either official SDK; the app still gets the star's reply byte-for-byte, plus two extra response headers.
Policy agreement asks whether your own application code would have branched differently, which a label-level benchmark cannot produce.
Reports include confidence-gated agreement and a swap verdict: the threshold that keeps disagreement inside a budget you choose.
Replay reruns recorded requests against a new checkpoint for free, keeping the star's recorded answers for comparison.

Watch out

Agreement is measured against the star, not against truth, so a local model that agrees with a wrong star is wrong too; only the labeled suites can say which one is right. The published launch report did not involve Jev at all: the machine could not hold the larger model, the star was a local model served in bfloat16, the double was a 421M model through a bundled sidecar, and there was no TypeSafe key available. Every verdict in that run was do not route. Option order is identical for every backend because they all receive the same body, so option-order sensitivity goes untested even though a separate benchmark measures it. Probability rounding differs between backends, which the validator tolerates proportionally, and latency is machine-specific and includes network time for hosted backends. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-column-race

Live race rig that labels 1,000 real app reviews with Jev typed questions against Gemini JSON output on identical batches, then replays recordings and re-ranks rows locally without extra model calls.