What does laya-jev-lab compare?

Jev, TypeSafe's closed API, against Laya, Convai's open-weight model, on the same idea: answer a typed question and return a probability instead of generating text. The repo adds a local-first cascade built on confidence thresholds.

How does the local-first cascade work?

Laya answers first in about 7.6 ms; if confidence is at or above the threshold the local answer is accepted, otherwise the question escalates to Jev at about 588 ms. At 0.60 on the 40-case benchmark that matched Jev's accuracy with 45% of traffic solved locally.

What hardware and software did the benchmarks use?

Everything ran on an Apple M4 Max with macOS 27, Python 3.14, and MLX 0.32.2. Each number comes from a script in the repo, with raw output published in results/.

How should the noul primitive be used?

The README's rule is to ask about facts, not judgements. Noul returned 0.996 on a message that clearly requests a refund, but near-zero probabilities for texts that were merely polite or appropriate.

Is the choice primitive sensitive to option order?

On arbitrary clipboard items it was correct across 23 of 24 orderings. On semantic categories it returned 0 of 24, but because it picked the same option every time, which the README reads as a task-design issue rather than ordering noise.

Can confidence be trusted to gate automation?

Only sometimes. The README's short answers are: only some primitives, only sometimes, and yes a cascade helps if traffic is mostly clear-cut. Raising the threshold past 0.70 cost latency without buying accuracy.

Who made laya-jev-lab?

It is yibie's independent attempt to compare typed-decision models, motivated by the fact that almost all published comparisons are vendor-supplied. The repo is MIT licensed.

Jev directory

CommunityJev Benchmarks & Evaluations

Laya Jev Lab

Independent measurements of typed-decision models: Jev (TypeSafe API) against Laya (open weights) on 40 Chinese support-ticket classifications, plus a local-first cascade that matches Jev's accuracy at about 1.8x the speed.

See highlights Open github.com

Stars: 13
Language: Python
License: MIT
Reading time: 2 min
Published by: Community
Author: yibie
Last verified: Oct 2026

yibie/laya-jev-lab

User repository on GitHub

View on GitHub

Independent measurements of typed-decision models: Jev (TypeSafe API) vs Laya (open weights), and a local-first cascade that matches Jev's accuracy at 1.8x the speed

Forks: 3
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

Tagscommunity catalog-import logicrw benchmark local-models mlx

Works forJev for Classification Jev for Evaluation Jev for Decision gating

Used inJev for Customer Support Jev for Software & DevTools

Languages: Python
Platforms: macOS
Capabilities: Open sourceRuns locally

Highlights

Jev scored 31/40 (78%) at 588 ms; Laya locally scored 23/40 (57%) at 7.6 ms on an Apple M4 Max, macOS 27, MLX 0.32.2.
At a 0.60 confidence threshold the cascade matched Jev's 78% accuracy while solving 45% of traffic locally, 1.8x faster than pure Jev.
Jev reached 100% on the 20 clear single-intent cases, often with confidence exactly 1.00; both models struggled on boundary inputs.
Noul was reliable on explicit facts, 0.996 on a clear refund request, but near zero for judgements like politeness; ask about facts, not judgements.
Choice passed 23 of 24 option orderings on arbitrary clipboard items, yet picked the same option every time on semantic categories.

Quickstart

python

first = laya.ask(state, question, criteria)
if first["confidence"] >= threshold:
    return first
return jev.ask(state, question, criteria)

Watch out

Copy reviewed against the repository README on 2026-10-15. Every number comes from the author's Apple M4 Max runs with raw output published in results/; the 40-case benchmark is small, and the README itself reports that Laya's confidence-to-accuracy curve is not monotonic.

Questions about this entry

Short answers drawn from the linked project or page.

From the Jev guides

Original write-ups on what this entry is for.

The trust ledger: measuring confidence your domain has earned

Oct 16, 2026

ASSAY-001 calibrated Jev on CLINC150 but found it overconfident on Banking77 — one audit, two verdicts. How calibration research measures the model-and-domain pair, discounts stated confidence, and signs the evidence.

Jev Benchmarks & Evaluations#calibration#evaluation#open-models

Read article

From the Jev AI model community

Posts from builders shipping with Jev right now.

Follow @typesafeai