Is this benchmark measuring Jev, or my network connection?

Both, inseparably. The clock wraps the awaited SDK call, so it covers local SDK work, network travel, server processing and response parsing, and excludes client construction, console output and writing results. The README also flags sample size, case order, connection reuse, server load and network conditions as things that move the numbers, and asks you to repeat runs and keep the raw reports.

How many paid calls does one benchmark run cost me?

Twelve with the defaults: one warmup plus five measured attempts for the one-question case, then the same for the three-question case, giving 2 * (warmups + iterations). Requests never overlap, the one-question case finishes before the three-question one starts, and the README notes calls may incur charges. Warmups and failed attempts are excluded from the reported statistics.

Can it leak my key or quietly spend it?

The key has to come from somewhere, and the README is direct about that: use Read-Host -AsSecureString promoted into the process environment, or copy .env.example and edit .env, but never pass a key as a CLI argument and never share .env. An absent, blank or unchanged placeholder key fails before any request goes out, and output redacts exact occurrences of the active key.

Will it pick up my host or model environment overrides?

No, deliberately. The CLI fixes the API host at https://api.typesafe.ai/v1/systemone, requests jev-latest, disables SDK logging and automatic retries, and uses a 30-second timeout unless you override it. It explicitly does not use SDK environment overrides for the host, model or log level, so your shell config will not silently redirect it somewhere else.

How do I set it up and run the examples?

Install Node.js 20 or newer and Git, then npm ci, npm test, npm run typecheck, npm run build, and npm start -- --help. The examples run with npm run smoke, batch, or benchmark, and the SDK dependency pins the official upstream v0.6.0 release tarball from GitHub with a recorded integrity hash.

What can I tune on the benchmark?

--warmups from 0 to 1000 (default 1), --iterations from 1 to 10000 (default 5), and --timeout-ms from 1 to 300000 (default 30000). The table reports successful sample count, median, mean, minimum and maximum latency, and an all-failed case shows count 0 with null statistics.

What does a run save, and when does it fail?

Every run writes results/benchmark- .json with configuration, timestamps, the synthetic state, per-attempt latency, phase, a response or safe error summary, and summary statistics. The CLI exits with status 1 if any attempt failed, including a warmup, and errors report HTTP status, timeout or connection failure without dumping bodies or headers.

CommunityJev Practices & Patterns

Jev Lab — Menny1337

A small Node.js TypeScript CLI for trying Jev on synthetic support text: one urgency question, three questions in a single request, or a latency comparison. No web server and no UI.

See highlights

Open github.com

Stars: 0
Language: TypeScript
Published by: Community
Author: Menny1337
Last verified: Oct 2026

Menny1337/jev-lab

User repository on GitHub

View on GitHub

TypeScript experiments, evaluations, and latency benchmarks for TypeSafe's Jev model

Forks: 0
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

This is one of several Jev listings for the same project. The canonical entry is Jev Lab — jammaru.

About this project

A small Node.js TypeScript CLI for trying TypeSafe's Jev model with synthetic support text. Run one urgency question, ask 3 questions in one request, or compare their request latencies. There is no web server or user interface. Install Node.js 20 or newer and Git. Prefer a currently supported Node.js LTS release. The official SDK requires Node.js 20 or newer.

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import awesomejev

Works forJev for Classification Jev for Evaluation

Used inJev for Software & DevTools

Languages: TypeScript

Highlights

batch sends urgency, department and frustration as one request, which is the cheapest way to see multi-question behaviour.
The benchmark runs the one-question case to completion before the three-question case, with warmups excluded from the statistics.
Every run writes a JSON report with config, timestamps, per-attempt latency, phase and the model the server actually returned.
Tests use mock transports and synthetic responses and never use a real key or call the API; CI runs on Node 20 and 24.

Watch out

The setup documentation is written for Windows PowerShell, which is unusual and will not map directly onto a Unix shell. The SDK dependency pins the upstream v0.6.0 release tarball from GitHub with a recorded integrity hash rather than coming from npm, so installs reach outside the registry. All commands request the jev-latest alias, and the README notes the underlying model can change under you; the CLI hardcodes the API host, disables SDK logging and automatic retries, and ignores SDK environment overrides for host, model and log level. A default benchmark run is 12 API calls and calls may incur charges. The timing is a monotonic clock around the awaited SDK call, so it includes local SDK work, network travel, server processing and response parsing while excluding client construction, console output and writing results — it is not isolated inference time, and small samples, case order, connection reuse and server load all move it. An all-failed case reports count 0 and null statistics rather than zero latency, an interrupted process can leave an empty reserved output file, and the exit status is 1 if any attempt failed including a warmup. Output redacts exact occurrences of the active key, but .env is gitignored rather than prevented, and the README asks you to inspect staged changes. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-lm

Word-level language model experiment that uses Jev as its output layer, with explicit tokenizer, sampler, repetition mask, stop rules, and chunk verification.