User repository on GitHub
Reproducible Jev vs Luna review-classification benchmark with measured accuracy, latency, and costs.
- Forks
- 0
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
CommunityJev Practices & Patterns
Jev Vs Luna
A reproducible English benchmark running one review-classification rubric through jev-latest and gpt-5.6-luna, publishing per-field accuracy, macro F1, latency and API-reported cost alongside the raw responses and run manifest.
- Stars
- 0
- Language
- Python
- Published by
- Community
- Author
- mameli
- Last verified
- Oct 2026
About this project
A small, reproducible English benchmark comparing typesafe/jev-latest through OpenRouter's Decisions API with openai/gpt-5.6-luna through its chat completions API. Both receive the same review text and the same English classification rubric: topic, sentiment, inferred stars, whether a reply is needed, and whether an actual product defect is reported.
Quoted from the project's README. Read the full README.
Works forJev for Classification
Used inJev for Software & DevTools
- Languages
- Python
Highlights
- Luna scored 97.13 percent overall field accuracy against Jev's 96.13 percent with failures counted wrong, a 1.00 point gap per model.
- Jev's median latency was 0.647 s against Luna's 1.556 s, and its mean cost per attempt with a known cost was about 4.9 times lower.
- Cost comes only from numeric API-reported usage.cost, never a hard-coded price, and a missing cost is recorded as unknown rather than zero.
- The test suite and the dry run need no credentials at all, and the charts rebuild from the published report with no API calls.
Watch out
Read the author's own framing first: this is one synthetic diagnostic run and not evidence of a general model ranking, and Luna scored slightly higher against the fixture labels. The dataset is 100 texts built from 50 explicitly labeled synthetic cases with two contextual variants each, variants share a case_id and are not independent samples, repetitions reuse the same reviews, and no confidence intervals are presented. Jev's cost total is incomplete rather than definitive: one request returned HTTP 520, took 41.03 seconds and reported no cost. The cost chart scales mean reported cost per call up to 1,000 calls and is not a separate 1,000-call experiment. Three cases where both models disagreed with the rubric, a wrong-size exchange case, an account-login case and a resolved crushed-box complaint, are read as label ambiguity rather than model failure, and the labels were not changed after seeing the outputs. Jev was requested as a floating alias and OpenRouter returned a dated build. Since the run, compare_models.py was renamed to benchmark.py and review_task.py to classification.py, while the measured results and dataset are unchanged. Copy reviewed against the repository README on 2026-10-04, not from running it.
Questions about this entry
Short answers drawn from the linked project or page.
No, and the README says so in the first paragraph: Luna scored slightly higher against the fixture labels, and this is one synthetic diagnostic run rather than evidence of a general model ranking. The structure limits how far it can be read: 100 texts from 50 synthetic cases, two correlated variants per case, repetitions over the same reviews, and no confidence intervals. The useful result is the cost and latency profile, not the accuracy ranking.
No. The cost chart scales the mean reported cost per call up to 1,000 calls, and the README states explicitly that this does not represent a separate 1,000-call experiment. Totals come only from numeric API-reported usage.cost in USD, missing cost is treated as unknown rather than zero, and the report carries a cost_complete flag plus coverage counts so you can see how much of the run is priced.
Failures count as incorrect for every field rather than being dropped, which costs Jev here: it returned 299 valid responses out of 300, the missing one an HTTP 520 in repetition 2 that counted as five incorrect fields and took 41.03 seconds. That is why Jev's end-to-end defect and all-five-fields-correct figures sit below what its valid-response accuracy implies, and why its reported cost total is incomplete.
Yes. uv sync, uv run pytest -q and uv run benchmark.py --dry-run all run offline, and the test suite makes no API calls even when a key is configured. The charts rebuild from the published report with uv run --group charts charts.py. A real comparison needs OPENROUTER_API_KEY, and the small paid smoke run is uv run benchmark.py --limit 3 --repeats 1 for 6 measured calls plus 2 warmups.
100 reviews, three repetitions, two models, plus one warmup each, for 602 completed attempts over about 12 minutes, with a shared rubric, a 60-second timeout, and no retries on either side.
Median latency on valid responses was 0.647 s for Jev against 1.556 s for Luna, and p95 was 0.910 s against 2.084 s, so Luna's median was 2.40 times Jev's.
data/reviews_100.json holds 50 synthetic cases with two contextual variants each, giving 100 distinct texts; the variants share a case_id and are not independent samples.
More Jev resources like this
24GitHub stars
Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
rorshopping/jev-on-a-laptopPython
rorshopping/jev-on-a-laptopPython22GitHub stars
Offline evaluator that checks whether classifier confidences match reality and converts mistake costs into a deployable threshold, emitting an HTML report plus machine-readable threshold files.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
rlaope/jevalPython
rlaope/jevalPython16GitHub stars
Vendor-neutral harness that tests whether Jev reranking improves a small RAG stack, using frozen hybrid BM25 plus BGE-M3 candidates and isolating retrieval, rerank, and generation stages.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
erendikmenn/jev-rag-benchmarkPython
erendikmenn/jev-rag-benchmarkPython37GitHub stars
Minimal Python recipe that turns any causal LLM into a Jev-style classifier by prefilling choice_index and constraining sampling to legal indexes in one forward pass.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
sgoedecke/system-onePython
sgoedecke/system-onePython39GitHub stars
Open replica of the typed decision API on openly available base models, answering choice, score, and true-probability questions in one forward pass with no decoding.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
mithalouni/system-one-openPython
mithalouni/system-one-openPython941GitHub stars
Type-safe generation layer over existing autoregressive models via JSON Schema on SGLang, returning constrained string, numeric, boolean, and enum answers with optional thinking.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
zmtomorrow/TypeARPython
zmtomorrow/TypeARPythonFrom the Jev guides
Original write-ups on what this entry is for.
Most Jev accuracy problems are state problems. What belongs in a state, what is a pre-made judgment to leave out, and why naming the slice you ask about matters more than the wording.
Jev Practices & Patterns#state#design#primitives
Read article
A Noul returns one number and no separate confidence. How to read the value, pin its boundary with criteria, split compound questions, and set a per-action threshold that matches the cost of being wrong.
Jev Practices & Patterns#noul#primitives#confidence
Read article
A Score is a weighted position, not a ruler. Why the number can land between levels, why levels are judged separately, and how to combine several atomic scores in code instead of one crowded rubric.
Jev Practices & Patterns#score#primitives#scoring
Read article
From the Jev community
Posts from builders shipping with Jev right now.