Question 1

Does this replace the official TypeSafe SDK, or sit on top of it?

Accepted Answer

On top. The README is explicit that it is not another wrapper: the official SDK already provides a typed client, and this project covers the layer applications still need around the model — decision contracts, policy thresholds, fallbacks, evaluation and production examples. It builds on the upstream systemOne() requests and the noul, choice and score shapes rather than duplicating transport or response types.

Question 2

If the model call fails, do I get a fallback route or an exception?

Accepted Answer

An exception. Provider and API failures and malformed SDK responses reject; the engine does not produce a DecisionOutcome in that case. The fallback route defined in a policy is for low-confidence answers, not for transport errors, so your own handling has to decide whether to retry, degrade to something else or surface the failure to the caller.

Question 3

How do I pick thresholds instead of copying the ones in the examples?

Accepted Answer

That is one of the stated design principles: evaluate before automating, and choose thresholds from task-specific data. The offline harness gives you the curves to do that — accuracy and confusion matrices, per-label precision, recall and F1, Brier score, weighted ECE, coverage and risk threshold sweeps, binary-band sweeps, and risk-coverage with AURC. pnpm eval:live needs a key and is opt-in only.

Question 4

Is the RAG evaluator a real evaluation or a demo?

Accepted Answer

It is the repository's first full showcase rather than a minimal classification demo. Retrieval and generation judgments stay separate, and accuracy, Brier score and weighted ECE are reported per component: chunk relevance, context sufficiency, evidence conflict, answer relevance, groundedness, contradiction, and reference-based correctness when a reference answer exists. The diagnosis is composed in TypeScript instead of one large question, with Jev-only, baseline-judge and cascade comparison modes.

Question 5

How do I install it and what does it need?

Accepted Answer

Node 20 or newer, then pnpm add jev-starter. The runtime package is ESM and exposes its public API from jev-starter. The published version is 0.1.1 on npm with provenance and a matching GitHub Release, and the README warns that main may already carry changes intended for a later release, so pin what you install.

Question 6

Is there anything to run before spending money?

Accepted Answer

Yes. pnpm eval:offline drives the harness over the labeled JSONL fixtures with no key and no network, producing accuracy, confusion matrices, per-label precision/recall/F1, Brier, weighted ECE, coverage and risk sweeps, binary-band sweeps, risk-coverage with AURC, latency, failures, usage, comparison and cascade reports. pnpm eval:live is the opt-in path and requires TYPESAFE_API_KEY.

Question 7

Where do two-sided thresholds and observability actually show up?

Accepted Answer

A policy of kind confidence names a question plus an autoThreshold and a fallbackThreshold, and the engine returns only the route: auto, fallback, or review. Binary judgments support two-sided thresholds so a low P(true) routes differently from an ambiguous result, and observability exposes model, latency, confidence and probabilities, the selected policy path, and the outcome as decision events and metric hooks.

Jev Starter — hamakyo

About this project

Highlights

Questions about this entry

Other listings of this project

More Jev resources like this

From the Jev guides

From the Jev community