Open Jevlike Infer

They are repo-reported comparisons on one A800 with full setup in benchmarks results clef-flash-a800.md: 45ms versus 60ms short p50, 75ms versus 177ms image p50, and 28 versus 15 req/s at concurrency 16. Re-run the reproducible benchmarks on your GPU before trusting them.

Yes for covered paths. It enforces Jev limits, handles images and videos as data URIs, base64, URLs, or frame lists, and returns choice, score, and noul answers with usage. Clients written for Jev work unchanged.

Plain vLLM serving cannot run the custom scoring head. This project runs the backbone as a vLLM pooling model in-process, checks tokens against the model encoding, and scores options with the model head across micro-batched requests.

Expect Linux, Python 3.10+, CUDA 12+ drivers, a sha256-verified weight download, and about 100 seconds of loading and warm-up before jevlike-infer serve answers on port 8000. Docker and systemd units are included.

One: Cloudflare's clef-flash, a 9B multimodal model, on the vllm backend by default or hf, validated on an A800 80GB with driver 550. More open Jev-like models are planned, and docs/adding-a-model.md explains how to add one yourself.

The token sequence vLLM produces is checked token by token against the model's own encoding, and answers match the reference transformers implementation to a maximum total-variation distance of 0.0067. Beyond unit tests in CI there is an 87-check end-to-end suite that runs against a live server, and the comparison table lists ARC-Challenge and ARC-Easy at 98.6 and 99.6 against the model card's 98.3 and 99.5.

Everything is a flag or a JEVLIKE_ variable: --api-key to require a bearer token, --gpu-memory-utilization at 0.5 by default, --max-batch and --batch-window-ms at 32 and 2 for micro-batching, --no-media-urls to reject http(s) media and accept only inline data, and --max-media-items 16 with --max-media-bytes 50 MB. It is Apache-2.0, with weights carrying their own licences.