Is this genuinely a drop-in replacement for the hosted API?

On the wire, yes. It serves POST /v1/systemone with the same request and response body as Jev, returns the same 422 validation list shaped {detail: [{loc, msg, type}]} and 401 on a bad key, and answers /v1/models. The difference is upstream: the answers come from an open Qwen3.5 checkpoint you serve locally, not from TypeSafe, and the model field reports the served id.

Can I trust the numerics against transformers?

The repository ships the check instead of asserting it: an oracle test runs real prompts through both engines and compares full-vocabulary logits at the read-out position, reporting identical argmax and top-5 across 0.8B, 2B, 4B and 9B. Against an fp32 reference it claims to sit closer to fp32 than the reference's own bf16 path, because the residual stream stays fp32 while GEMMs run in bf16.

Is the speedup coming from the kernels or from caching the shared prefix?

Both are separated in the repo's benchmark, which fixes a 1,024-token state and 16 questions of 40 tokens. The shared-prefix path is measured against a naive engine that re-prefills state and question for every question, and against batched HF transformers with flash-linear-attention installed, plus a row for when the state is already cached. Responses carry prefill, eval and prefix-cached timing headers.

Which model sizes can I actually run?

Only the dense text sizes in the README's table, and of those 27B is marked not yet working because six query heads per KV head needs a kernel variant, while MoE checkpoints are unsupported. Workspace is sized for 8,192 state tokens, 4,096 branch tokens and 256 branches by default, and shrinks with the max-token flags. The 4B checkpoint is the recommended one.

What hardware and toolchain does it need?

Linux with an NVIDIA GPU of compute capability 8.0+ (Ampere or newer), CUDA toolkit 12.4+ or 13 with nvcc, CMake 3.24+, and uv; set CUJEV_CUDA_ARCH for your card, such as 120 for an RTX 5090 or 86 for an RTX 3090.

How do I install and serve a model?

CUJEV_CUDA_ARCH=86 uv sync builds libcujev.so, then uv run scripts/download_model.py Qwen3.5-4B fetches the checkpoint, and uv run cujev serve --model models/Qwen3.5-4B --port 8080 starts the server.

What timing headers does the server return?

Responses carry x-cujev-prefill-ms, x-cujev-eval-ms, and x-cujev-prefix-cached, and GET /v1/models and GET /health are also available for discovery.

CommunityJev Practices & Patterns

cu-Jev

A CUDA-native decision engine that turns an open Qwen3.5 checkpoint into a System One endpoint with Jev's wire format, answering many typed questions per state from C and CUDA with no PyTorch at runtime.

See highlights

Open github.com

Stars: 5
Language: C
License: APACHE-2.0
Published by: Community
Author: dtunai
Last verified: Oct 2026

dtunai/cu-Jev

User repository on GitHub

View on GitHub

cuda-Jev — a CUDA-native Jev System One decision inference engine. Jev compatible API, examples, and reproducible benchmarks.

Forks: 0
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.Homepage

About this project

cu-Jev A CUDA-native System One decision engine with a Jev-compatible API. One state in, many typed decisions out. Full C/CUDA, a little Python, no PyTorch at runtime. POST /v1/systemone {state, model, questions} → {model, answers, usage}. cu-Jev turns an open Qwen3.5 checkpoint (0.8B · 2B · 4B · 9B) into a decision function with the exact wire format of TypeSafe's Jev: send a state and a map of typed questions (noul / choice / score), get typed answers with probabilities and confidence. Nothing is decoded, ever.

Quoted from the project's README. Read the full README.

Tagscommunity awesomejev catalog-import c

Works forJev for Classification Jev for Evaluation

Used inJev for Software & DevTools

Highlights

Nothing is decoded: at the last token of each question branch it reads the option-label logits straight off the LM head.
The state runs once; each question is a short branch over its KV cache, and branches never see each other.
Point TYPESAFE_BASE_URL at it and the official Python and JS SDKs work unchanged, models.list() included.
Oracle tests compare full-vocabulary logits against transformers at the read-out position: identical argmax and top-5 on every probe.

Watch out

The hardware envelope is narrow: Linux, an NVIDIA GPU of compute capability 8.0 or newer, CUDA 12.4 or 13 with nvcc, CMake 3.24+, and a build step. The README lists Qwen3.5-27B as not yet supported because six query heads per KV head needs a kernel variant, all MoE sizes are out, and the vision tower and MTP head inside the checkpoints are skipped. Speed figures are its own on one machine. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-on-a-laptop

Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.