Are the reported accuracy numbers independently verified?

They are zero-shot controls on 231 public JevBench items with temperatures fitted on the mix calibration split, never on JevBench; 4B reaches 0.779 accuracy with 0.062 ECE.

How is this different from hosted Jev?

It is an independent implementation not affiliated with TypeSafe AI or JevBench maintainers, returning distributions from letter logits at a single answer slot from a state plus bounded rubric.

What breaks if I change quantization after calibrating?

Calibration is bound to the checkpoint and, on MLX, to the quantization it was fitted on, so refit temperatures after converting or quantizing.

Can I run tests without weights or a TPU?

Yes. The CPU suite runs with hardware- and weight-dependent tests marked, using 4 virtual devices; TPU compile-check, training, and Kaggle runners need hosts and weights.

What license does tjev use, and can I build on it commercially?

Apache-2.0, and the Qwen3.5 weights are Apache-2.0 too. Each mix's manifest records the license of every data source, and some sources are non-commercial CC BY-NC, which tjev data build --commercial-only excludes, so check the manifest before shipping.

What hardware does it train and run on?

Training is JAX, Flax NNX, Optax, Orbax and Grain with Pallas TPU kernels; inference can export to MLX for Apple silicon or serve a FastAPI decision API. It needs Python 3.12+ and uv, and uv sync picks a dev environment with CPU JAX, tests, parity and data.

How does it guard against benchmark contamination?

The mix-v3 recipe combines contenders' public JevBench-style data, code-labelled EN/FR generators, public datasets recast as rubric decisions, and a small text-analysis block, and every training row is checked against the public JevBench items for contamination; held-out generator sets are kept separate.

CommunityJev Tools & CLIs

Tjev

Independent Qwen3.5 typed-decision stack in JAX with Pallas TPU kernels, LoRA tuning, per-type calibration, and MLX export.

See highlights

Open github.com

Stars: 0
Language: Python
License: APACHE-2.0
Published by: Community
Author: tychota
Last verified: Oct 2026

tychota/tjev

User repository on GitHub

View on GitHub

Calibrated typed decisions from Qwen3.5 label logits: LoRA fine-tuning in JAX / Flax NNX / Optax / Orba. x / Grain, with Pallas TPU kernels, and export to MLX for Apple silicon. This is the JevBench "Jev-class" task. Tev is an independent implementation, not affiliated with TypeSafe AI

Forks: 0
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

Calibrated typed decisions from Qwen3.5 label logits: LoRA fine-tuning in JAX / Flax NNX / Optax / Orbax / Grain, with Pallas TPU kernels, and export to MLX for Apple silicon.

Quoted from the project's README. Read the full README.

Tagscommunity github

Works forJev for Classification

Used inJev for Software & DevTools

Languages: Python

Highlights

Flax NNX decoder matches HF logits within 2.6e-5 in fp32 on 0.8B; bf16 top-1 stays 100% across 0.8B, 2B, 4B.
Per-question-type temperatures bound to checkpoint and quantization; training rows filtered against public JevBench items.
Mix-v3 blends public JevBench-style data, EN/FR generators, and recast sets; TPU plan targets free Kaggle and spot runs.
Serves /v1/systemone via FastAPI micro-batching with JAX and MLX backends; CPU suite uses 4 virtual devices.

Watch out

Pallas kernels have not compiled on a real TPU yet and fall back to XLA paths; see docs/results.md for measured training results. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-deep-dive

Long-form Jev writeup that maps each conclusion to official docs, vendor evals, integrations, community reproductions, and press, separating contract facts from unverified claims.