stephanj/parallelconstraintdecoding
User repository on GitHub
Parallel Constraint Decoding using Java and Llama.cpp compared to Python 🔥
- Forks
- 1
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Repos & SDKs
Parallel Constrained Decoding
A Java and Python implementation of parallel constrained decoding: fill a whole JSON schema of booleans and enums in two forward passes, with calibrated confidence per field and output that is schema-valid by construction.
- Stars
- 15
- Language
- Java
- License
- APACHE-2.0
- Reading time
- 3 min
- Published by
- Community
- Author
- stephanj
- Last verified
- Oct 2026
- Capabilities
- Open sourceRuns locally
Highlights
- On an Apple M1 Ultra the Java engine fills a 28-field schema in 266 ms cold and 130 ms warm, versus 2388 ms token-by-token and 337 ms in Python.
- Java drives llama.cpp through the JDK Foreign Function & Memory API with no JNI, loads any GGUF including local Ollama models, and ships a racing web app.
- Warm single-pass mode keeps the schema's KV cells between calls, a production optimization the Python reference engine does not have.
- Presets cover fraud/AML triage, code-security audit, support triage, a 255-choice tariff router, spam/phishing triage and Devoxx talk routing.
- The Python port runs the original MLX engine on Apple Silicon only with stock Qwen2.5-1.5B-Instruct; its Hugging Face source repo holds no weights.
Quickstart
bash
git clone https://github.com/stephanj/parallelConstraintDecoding.git
brew install llama.cpp
mkdir -p models
curl -L -o models/qwen2.5-1.5b-instruct-q8_0.gguf https://huggingface.co/Qwen/Qwen2.5-1.5B-InstructWatch out
Grounded in the truncated repository README, video link and GitHub metadata reviewed on 2026-10-15; all benchmark numbers come from one Apple M1 Ultra with an otherwise idle GPU, and on Linux the engine runs on CPU with seconds per race instead of sub-second.
Questions about this entry
Short answers drawn from the linked project or page.
Instead of generating a JSON object token by token, the technique fills a whole schema of booleans and enums in two forward passes, with calibrated confidence per field and output that is schema-valid by construction.
On an Apple M1 Ultra it filled 28-field schemas in 266 ms cold and 130 ms warm, against 2388 ms token-by-token and 337 ms for the Python parallel engine. The 255-choice tariff preset ran 158 ms cold and 67 ms warm.
Directly through the JDK Foreign Function & Memory API (no JNI). It runs on Metal on Apple Silicon and on CPU on Linux, and it can load any GGUF model, including models already downloaded by a local Ollama install.
A port of the original engine published at harshatheg/Qwen-2.5-1B-RLCD, kept for reference and comparison. It is MLX-based and Apple Silicon only, and despite the name that Hugging Face repo contains no weights or fine-tune, just source running stock Qwen2.5-1.5B-Instruct.
The presets show the intended shapes: fraud and AML triage, code-security audit, incident and support triage, a 255-choice tariff router, spam/phishing triage, and Devoxx CFP talk routing over a bank of 100 real talks.
macOS on Apple Silicon is recommended (Linux x86-64 or arm64 also works), plus JDK 22 or newer, Maven 3.9+, the llama.cpp shared library v0.4.1, and a GGUF model such as Qwen2.5-1.5B-Instruct Q8_0 (1.9 GB). Python 3.10+ and uv are only needed for the reference port.
Cold is the head-to-head: same prompt tokens and a full prefill on every call, the same timing boundary as the Python engine. Warm keeps the schema's KV cells between calls in the Java single-pass mode, which the Python engine does not implement.
More Jev resources like this
A correctness-validated Laya runtime for Apple Silicon that runs upstream Laya on the MLX GPU and the Apple Neural Engine, using the ANE only after an artifact passes a parity gate on your machine, plus a local Jev-compatible server.
Jev Repos & SDKs#community#catalog-import#madewithjev
Community
tc3oliver/laya-applePython
tc3oliver/laya-applePythonAn independent reimplementation of the OpenJev mechanism on PrismML's 1.75-bit Bonsai weights: typed decisions read straight off the next-token distribution of a 5.95 GB 27B model in one forward pass.
Jev Repos & SDKs#community#catalog-import#cobanov
Community
nicolailassen/open-bonsai-jevPython
nicolailassen/open-bonsai-jevPythonSelf-hosted Python SDK and System One-compatible API for runtime-defined Choice, Boolean and Score decisions on pinned open models, across Torch, vLLM, MLX, llama.cpp and WebGPU. Each result records probabilities, token usage, timing, model revision and prompt version, with committed row-level benchmarks.
Jev Repos & SDKs#community#sdk#self-hosted
Community
chengyongru/fastjevPython
chengyongru/fastjevPythonA tuner that trains language models to answer closed-set decisions with a complete probability distribution in one forward pass, using a tokenized multi-class Brier loss so only the correct choice is needed to train.
Jev Repos & SDKs#community#catalog-import#madewithjev
Community
liushiliushi/jevtunerPython
liushiliushi/jevtunerPythonRun Laya locally on Bun with native ONNX inference and Hugging Face tokenizers: an HTTP server implementing TypeSafe's decision API plus an English/Persian playground with 16 demos, usable through the official @typesafe-ai/sdk with a custom baseURL.
Jev Repos & SDKs#community#catalog-import#logicrw
Community
ali-master/usejevTypeScript
ali-master/usejevTypeScriptA native Swift implementation of OpenJev, the Jev-compatible typed-decision server: send a state and typed questions over a Jev-compatible HTTP API and read every answer from model probabilities, on Apple silicon Macs or inside iPhone and Mac apps.
Jev Repos & SDKs#community#catalog-import#cobanov
Community
algorythm-canada/openjevswiftSwift
algorythm-canada/openjevswiftSwiftFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.