Question 1

Is the 17.3 ms example figure a real end-to-end request time?

Accepted Answer

Not end-to-end. It times the forward pass over all 64 layers plus the scoring head, after prompts are built, tokenized and copied to the GPU once beforehand, and it excludes HTTP. The end-to-end figure the README reports is JevBench mean per-task latency of 42.0 ms against 258 ms for Open-Jev with flash-linear-attention, measured client-side with tokenization, HTTP and JSON included and a new connection per task, client and server on the same machine.

Question 2

What is the 6x actually measured against?

Accepted Answer

Two baselines. The first column is Open-Jev's default install, which does not ship flash-linear-attention, so Transformers runs the linear-attention layers as plain PyTorch ops at 256.0 ms. The second is Open-Jev with FLA installed, the same configuration used for the JevBench baseline server, at 103.7 ms. Both servers ran with `--max-length 16384` so every JevBench task would fit.

Question 3

Do the hand-written CUDA kernels change the model's answers?

Accepted Answer

On one task out of 231. JevBench accuracy was 197/231 for open-jev-fast against 198/231 for the original server in the authors' own runs, which the README attributes to a single near-tie prediction changing. The Open-Jev-27B-v1.1 model card reports 197/231 for this checkpoint, and the kernels are written to reproduce the bf16 rounding points of the Transformers and FLA references.

Question 4

What happens the first time a new request shape arrives?

Accepted Answer

It pays a capture cost. CUDA Graphs are cached per row bucket, candidate count and path-length bucket, up to 512 graphs, so a new shape means capturing another. On a first pass after start the README reports 190 ms mean per task with a 3.3 s slowest case for this server, against 1967 ms mean and a 65 s slowest case for the original compiling FLA's Triton kernels for each new shape.

Question 5

What hardware does the benchmark use?

Accepted Answer

One NVIDIA B300 in bf16 with no quantization; the example-request numbers come from a single script run on the same GPU, and both servers were measured on a second pass.

Question 6

What does the prefix tree do?

Accepted Answer

One packed forward pass computes the shared context once, each question's text once, and each candidate's tail, so the weights are read once; for the example request computed rows drop from 574 to 277.

Question 7

What does it need to run?

Accepted Answer

An Open-Jev installation and the model weights, since it is a drop-in backend that keeps the same request format and the same Open-Jev-27B-v1.1 checkpoint on Qwen3.8-27B.

Open Jev Fast

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev guides

From the Jev community