Jev Persian Benchmark

It benchmarks Jev, Jeff, Mizan and Laya on Persian questions: 480 authored general questions, 560 literary questions, and a classical poetry pilot of 24 excerpts with 48 main questions plus 48 controls. Main metrics exclude diagnostics, and the general benchmark includes a historical Laya comparison.

Across five runs on dataset v1.0.0, Jev recorded 239/240 (99.6%) choice accuracy and 159/160 (99.4%) Noul accuracy, with 624/624 valid answers across 106 requests per run and no failures. Its six errors concerned exclusive availability, permissions, unrecorded consent and a cosmetic button change.

The README reports 24/24 on meaning recognition and 24/24 on situation matching for the classical-pilot run. Worked examples and run results are in the repository's docs/classical-poetry.md, showing exactly what Jev received and selected.

Jev via jev-1.13.0 and typesafe-sdk 0.7.1; Jeff at Qwen3.5 0.8B and 2B on an Apple M4 with the MLX backend in BF16; Mizan v3 Q8_0 GGUF on an Apple M4 with llama.cpp b11600 and Metal; and Laya multilingual in fp32 on an Apple M4 CPU with PyTorch 2.14.0.

No. The README says raw responses are saved and answers are scored locally, without a runtime model judge. Related general questions are batched while poetry questions run individually, and main metrics exclude diagnostics.

On matched pairs, Jev held 24/24 on both invariant and contrast cases, while Laya dropped to 14/24 invariant and 16/24 contrast. English instructions changed no Jev decisions out of 48, and repeatability across 48 three-observation groups showed no decision changes for any model.

The repository is maintained by armanjr and written in Python. The Jev runs used the TypeSafe API through typesafe-sdk 0.7.1, so reproducing a live Jev run would require a TypeSafe key; the local comparisons used MLX and llama.cpp on Apple silicon. Per-model checkpoint and runtime details are in the repo's guides.