User repository on GitHub
- Forks
- 0
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Benchmarks & Evaluations
JEV Fraud Risk POC
A local Java benchmark that plays 10,000 deterministic transactions through two fraud pipelines: every one to Qwen via Ollama, or through System One/Laya first, where a low-risk answer at confidence 0.70 or higher stops before the chat model.
- Stars
- 0
- Language
- Java
- Reading time
- 2 min
- Published by
- Community
- Author
- AliveMonster
- Last verified
- Oct 2026
Highlights
- Two architectures share one 10,000-transaction dataset: all-Qwen, versus a System One/Laya gate that stops low-risk at confidence 0.70.
- Dataset: 1,000 customers, 9,000 safe and 1,000 fraudulent transactions, seed 42, multiple fraud patterns and deliberate anomalies in safe rows.
- actualFraud and fraudPattern are benchmark ground truth and must never be sent to either model.
- Needs Java 17+, Maven 3.9+, Ollama for Qwen and a Laya endpoint for architecture 2.
- The generated fraud-dataset.json is checked in so benchmark runs stay reproducible.
Quickstart
bash
ollama pull qwen3:1.7b
ollama serveWatch out
README reviewed 2026-10-17; the repository lists no license and the captured text stops mid-command, and the digest includes no result tables, so which architecture wins on cost, accuracy or latency is not stated here.
Questions about this entry
Short answers drawn from the linked project or page.
A local, end-to-end comparison of two fraud pipelines on the same deterministic dataset: Architecture 1 sends every transaction to Qwen through Ollama, while Architecture 2 routes through System One/Laya first.
In architecture 2, every transaction goes to System One/Laya; a LOW result with confidence at or above 0.70 stops there, and anything else continues to Qwen. The repository root is named JEV and Laya is the local decision model it benches.
1,000 customers and 10,000 transactions: 9,000 safe, 1,000 fraudulent, deterministic with seed 42. It includes multiple fraud patterns, and some safe transactions intentionally contain anomalies.
You need Java 17+, Maven 3.9+, Ollama and Laya. Pull and start Ollama with ollama pull qwen3:1.7b and ollama serve on localhost:11434, keep the checked-in fraud-dataset.json for reproducibility, then run the Maven application.
No. actualFraud and fraudPattern are benchmark ground truth and the README says they must never be sent to either model; they exist to score how each architecture performs.
Yes. The repository already contains a generated dataset, and the README gives a Maven package command plus a Java invocation of com.example.fraudpoc.dataset.FraudDatasetGenerator to rebuild it deterministically.
The captured README describes the setup but includes no result tables, benchmarks or conclusions, so the repository's actual outcome is not documented here; it is a benchmark harness rather than a published study.
More Jev resources like this
Independent measurements of typed-decision models: Jev (TypeSafe API) against Laya (open weights) on 40 Chinese support-ticket classifications, plus a local-first cascade that matches Jev's accuracy at about 1.8x the speed.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
yibie/laya-jev-labPython
yibie/laya-jev-labPythonThe code and data behind ProbabilityBench: text-based simulated worlds with exact class probabilities, so a decision model's reported probabilities can be compared directly with ground truth. An arXiv paper and a Hugging Face dataset accompany it.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
ishida-lab/probabilitybenchPython
ishida-lab/probabilitybenchPythonA reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.
Jev Benchmarks & Evaluations#community#python#evals
Community
keta1930/what-the-jevPython
keta1930/what-the-jevPythonA deterministic gridworld that benchmarks System One decision backends, typed calibrated decisions over a declared option set with one forward pass and no decoding, in a sequential loop, with a scripted oracle scoring agreement without labels.
Jev Benchmarks & Evaluations#community#github#benchmark
Benchmark harness that scores TypeSafe Jev against any OpenRouter model or local checkpoint on labelled classification data: accuracy, calibration (ECE), confidence distribution, latency and cost across five tasks with 300 items each.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
4esv/jev-evalPython
4esv/jev-evalPythonThe benchmark behind JevAgentHarness's paper on decision models that share control with LLMs: next-tool decisions scored at isolated, open-loop and closed-loop levels, with 3,939 common-state questions and 153,621 answers from 39 models. The L1 question texts are third-party data and are not redistributed. CC BY 4.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#agents
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.