Laya Fast

Laya is Convai Innovations' 421M ModernBERT-large typed-decision model (Apache-2.0) that answers a constrained question about text with a choice, a rubric score or a yes/no probability, without generating tokens. Laya Fast is an independent Apple Silicon runtime for it: MLX GPU inference plus optional Core ML execution on the Neural Engine.

A single short question (≤128 tokens) runs on the Neural Engine when its compiled body is present; a batch is split so a worker runs some questions on the ANE while the GPU batches the rest, using a measured cost model (ane/cost_model.json). A single 512-token question ties across engines and stays on the GPU, as does anything longer than the largest compiled bucket.

Nothing breaks: missing ANE bodies are not an error, and LayaFast degrades to the same GPU-only path as `--agent mlx`, which the README calls the first-run route that is always available after conversion.

On the author's M3 Max (30 GPU cores, 36 GB, fp16): 9.7 ms for one 74-token question vs 13.9 ms MLX-only, 41 ms for 8 short questions vs 65 ms, 54 ms for one 512-token question (a tie), and 219 ms for 8×512 vs 374 ms. A 2026-09-23 confirmation run measured 9.57 / 41.73 / 54.23 / 210.76 ms p50.

They are single-machine fixture medians, not cross-device results, and your numbers depend on chip, memory and load. The 8×512 win is a 4/4 split across two engines rather than a faster kernel, and it requires the compiled ANE bodies. Process footprint was about 1.5 GB with the MLX buffer cache capped at 128 MB.

No on both counts. It is not an official Convai release; it ports the Apache-2.0 reference implementation, and the ANE file adapts mizorewww/laya-coreml's prototype. The quality suite uses synthetic labels created for the benchmark, so it is a smoke test, not production accuracy or a general comparison with Jev.

`--agent mlx` is the CLI default, GPU-only and always available after conversion; `--agent fast` opts into the ANE + GPU router. Because LayaMLX and LayaFast share the same request and response shape, you can switch without changing your integration code.