Quantization is sold as a memory saving, but a decision model's product is a distribution. Compressing the backbone can leave the top answer unchanged while moving probabilities enough to break a confidence gate. The open artifacts show both outcomes: small KL divergence with a measurable top-1 drop, and a 0.210 probability shift on a four-example check. The head stays in higher precision even when the language weights do not.
What actually gets compressed
Most builds quantize the language backbone and leave the head alone. JEV-35B AWQ INT4 uses INT4 weights with FP8 activations on routed experts and INT8 elsewhere, while the 264-slot decision head stays bf16 and the vision tower, routers, and shared-expert gate are untouched. It serves on one 24 GB GPU via vLLM at 57.3 ms median and 17.44 requests/s at concurrency 1. JEV-27B-VL AWQ does the same for a 27B multimodal build: every text linear is INT4 with INT8 per-token activations, the rank-32 LoRA decision head stays a LoRA, and the bf16 vision tower is spliced back after quantization.
The head can also be the only thing kept in full precision. Jev-Omni Q4_K_M is a 6.87 GiB backbone at 4.95 bits per weight, plus a 116.38 MiB projector and a 3.78 MiB FP32 decision head. Measured on an RTX 5080 Laptop GPU, it uses 8,980 MiB of VRAM text-only, 9,148 MiB with the projector, and 9,256 MiB after an image decision. Laya-MLX defaults to FP16 with FP32 available for closer numerical agreement, and kev-4b-mlx-fp16 ships a merged FP16 backbone with FP32 DeltaNet decay, gated-norm weights, and pointer head, plus a SHA-256 manifest for every file.
The exception is a conversion that drops the decision path. JEV-9B Core AI W8 is 9.5 GB of W8A16 assets for Apple silicon, but contains only the Qwen3.5-9B text backbone: no adapter, no 24-slot decision head, no calibration. It cannot make calibrated typed decisions. Context shrinks to 2,048 positions via a K/V ring cache, and the Python host runs 6 tokens/s on an M5.
Why confidence drifts
The head's output is a softmax over option scores. Quantization changes the hidden states and therefore the logits; a temperature fitted before compression may no longer match the new distribution. The published checks show the size of the effect.
| Artifact | Precision | Evidence on the quantized build | Calibration caveat |
|---|---|---|---|
| JEV-35B AWQ | INT4 weights, FP8/INT8 activations, bf16 head | KL mean 0.03, ECE 0.1548, choice top-1 0.8596 on 500 held-out rows | No comparison against unquantized JEV-35B |
| JEV-27B-VL AWQ | INT4/INT8, rank-32 LoRA head, bf16 vision | KL 0.0147 vs 0.0143 for W4A16; top-1 0.825 vs 0.851 | Baseline is another quantized build |
| Jev-Omni Q4 | Q4_K_M, FP32 head | 624/624 Persian answers; 3 of 4 text checks matched the winner | Maximum probability difference 0.210 |
| pplx-decider EXL3 | 4.00 bpw, 15.35 GB | 43/43 sanity checks; 30/30 stable under reversed order | No BF16-versus-EXL3 accuracy or parity study |
| JEV-9B Core AI | W8A16 | Quality checked on three short texts | Excludes the decision head and calibration |
The numbers are not directly comparable, but the pattern is. JEV-27B-VL loses 2.6 points of top-1 choice accuracy against its W4A16 baseline while keeping KL close. JEV-35B reports a low mean KL of 0.03 and ECE 0.1548 on 500 rows, a calibration number you should not inherit without refitting. Jev-Omni Q4 matches the winning option on three of four text checks, with a maximum probability difference of 0.210. None compares against the unquantized source on the same rows.
pplx-decider-v1-27b-EXL3 is the most candid. It is a 4.00 bpw conversion in 15.35 GB of safetensors, down from about 48.6 GiB of BF16. On an RTX 4090 it measures 135 ms at 128 input tokens, 725 ms at 2,048, and 2.777 s at 8,192, with 12.67 to 13.89 GiB peak PyTorch memory. Its sanity suite reports 43/43, including 9/9 routing, and reversed option order kept the same choice 30/30 times. The card still states that no BF16-versus-EXL3 parity study has been run. It also fails under TabbyAPI with AssertionError: Unknown architecture Qwen3_5Model, because the custom readout is not a normal LM head.
Apple silicon and CPU paths
Laya-MLX is the cleanest Apple path: Apache-2.0, FP16 by default, FP32 available, and a reported 13.4 ms median for a short English decision and 7.4 ms on the multilingual checkpoint on an M3 Max. It matched upstream's selected answer on 63 of 63 validation questions in FP32 and FP16, 378 of 378 comparisons: fidelity on fixtures, not accuracy on your data. The compile, padding, and prompt-cache options are opt-in.
kev-4b-mlx-fp16 is a packaging artifact rather than a new model: the backbone and adapter are merged in FP16, the pointer head and sensitive normalization weights stay FP32, and package.json records every file's SHA-256 and size. It publishes no benchmarks, so quality comes from Kev-4B's own card. The value is reproducibility: you can verify the download end to end.
The Omni MLX build is the most complete Mac example. Jev-Omni MLX 4-bit uses 4-bit affine language weights with group size 64, keeps the vision tower in BF16 and the 256-way decision head in FP32, and measures about 7.0 GB peak Metal memory on a 16 GB M4 Mac mini. It reports a median 963 ms text decision and 994 ms image decision, 87.88% micro accuracy and 85.90% group-macro on the public JevBench split, with raw ECE-10 of 0.04497. The shipped temperature file (T=1.11517) cuts public ECE to 0.03069 but worsens the held-out split from 0.06261 to 0.06774, which the card calls a post-hoc artifact rather than a guarantee.
What to measure after compression
Compare the quantized artifact with the unquantized checkpoint on the same held-out rows, not with another quantized build. Report top-1 agreement, KL divergence, Brier score, expected calibration error, and coverage at your operating threshold. Then slice by question type: JEV-27B-VL's image path checks out at p=0.99999 on a red-circle test, but that says nothing about a rubric score on a long document.
Check the runtime separately. The pplx sanity suite passes 43/43 while TabbyAPI startup fails, and JEV-9B Core AI runs at about 6 tokens/s because the Python host dominates. A model that cannot be served is not cheaper. Check context too: JEV-9B Core AI drops to 2,048 tokens, and Jev-Omni's video, image, and audio routes passed only smoke checks.
Finally, refit the calibration. The confidence page notes that confidence is a derived statistic and the full probabilities come back, so you can compute your own measure; Noul answers carry no separate confidence at all. The thresholds in the examples (0.5 for human review, 0.9 for auto-approval) are starting points, not constants. The jaggedness list adds two structural reminders: complementary probabilities can sum past 1, and the same question can disagree across primitives. A quantized model can make both worse.
Serving choices
If the goal is a 27B or 35B decision model on one 24 GB GPU, the AWQ W4A8 builds are the documented route: JEV-35B at 57.3 ms median and JEV-27B-VL at 1.76 s cold prefill for a 5,098-token decision. If the goal is a Mac, MLX Omni or Laya-MLX are the paths, with the caveat that Omni's temperature file does not generalise. For a CPU-only box, a BitNet student or GGUF build is the right shape, but verify the decision head is present and in FP32 before benchmarking.
Exporting decision models covers the GGUF and ONNX mechanics, calibration beyond ECE covers the metric choice, and Jev at home covers the local deployment context. Read quantization for the precision vocabulary, temperature scaling for the simplest correction, and expected calibration error for what the headline number does and does not capture.
