Jev directory
CommunityJev Benchmarks & Evaluations
Jev and Kev SMS Spam Hands-On
A five-notebook Jupyter hands-on that runs Jev and Kev, an Apache-2.0 decision model served locally with llama.cpp, over the SMS Spam Collection, comparing accuracy and threshold effects across Kev-4B, 9B, and 27B.
- Stars
- 0
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Five notebooks cover Jev basics, Kev on Mac, spam scoring, threshold tuning, and Kev model sizes.
- Spam comparison scores 130 test messages with about 390 Jev calls; threshold tuning uses 100 validation messages.
- Kev is an Apache-2.0 open decision model answering the same API shape, run locally through llama.cpp.
- Jev notebooks cost roughly 1 yen or less each; the Kev notebooks make no Jev calls and are free.
- Kev weights are 2.8GB for 4B, 6.4GB for 9B, and 15.8GB for 27B, downloaded into models/ on first run.
Questions about this entry
Short answers drawn from the linked project or page.
Jev is TypeSafe AI's cloud decision model and is paid. Kev is an Apache-2.0 open decision model that answers with the same API shape and runs on the Mac through llama.cpp, so the same code can be pointed at either.
The Jev notebooks are cheap: part1 makes about 20 calls for under 1 yen, part3 makes 390 calls for about 1 yen, and part3b makes 100 calls for under 1 yen. The Kev notebooks make no Jev calls and are free. TypeSafe has no free tier, so prepaid credits are needed.
A Mac with Apple Silicon, tested on an M5 Pro with 48GB of memory; 32GB or more is suggested for Kev-27B. You also need Python 3.13 or later with uv, VS Code with the Jupyter extension, and llama.cpp 0.6.0 or later.
Jev is a cloud service and submitted content is processed on servers in the United States, so personal or confidential data should not be sent. Kev runs locally and does not move data off the Mac.
Clone the repository, run uv sync, copy .env.example to .env with chmod 600, and write TYPESAFE_API_KEY into it. Open the notebooks in VS Code and select the folder's .venv as the kernel, then run cells from the top.
The SMS Spam Collection from UCI Machine Learning Repository (Almeida and Gomez Hidalgo, 2011, CC BY 4.0) is not bundled; the part3 notebook downloads it. data/test_ja.json holds 30 test messages translated into Japanese with the original labels.
No. Decision model answers can vary with the model version and between runs, so the expected outputs shown are the values at creation time. The code is MIT licensed, and the data follows CC BY 4.0.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Standard One SH (StandardThinking)
LinkedIn: What multimodal decision models actually change, and what they do not.
LinkedInJevDirectory.org
Hacker News: Decision Index 0.2.1 scores open reproductions of Jev on a frozen suite.
jackbravo4 pointsDecision Index 0.2.1 – 70 open reproductions of TypeSafe Jev's Decision ModelRead the thread on Hacker NewsHacker News: Pointer to a DiffusionGemma plus vLLM tutorial for a Jev-shaped classifier.
1 pointHow to build a Jev-style classifier with DiffusionGemma and vLLMRead the thread on Hacker NewsMore Jev resources like this
Hugging Face dataset holding the scored public predictions, latency timings, and environment for the Aster-JEV-27B vision run: 9,867 of 12,210 requests correct, public index 76.7057, mean 507.2 ms per request on one A800 80GB.
Jev Benchmarks & Evaluations#evaluation#benchmark#open-models
Community
Pytest-based framework that judges mocked LLM outputs with Cloudflare clef-flash, a 9B decision model served locally by Ollama. Metrics turn typed probabilities into values between 0 and 1 and PASS/FAIL against a threshold.
Jev Benchmarks & Evaluations#evaluation#local#python
Benchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
Independent experiment scoring 400 real job postings against 30 fictional résumés with Jev, producing 12,000 evaluations in about 197 seconds for roughly $0.17, with a 100-posting answer key and an analysis of what the model got wrong.
Jev Benchmarks & Evaluations#github#madewithjev#evaluation
Communitykaucap08/jev-job-matching
Controlled benchmark of Jev against Claude Sonnet 5 on one coding-agent decision: given a bug report, which file needs editing? It scores 200 real SWE-bench Verified issues with cost, latency, and confidence-band routing tables.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
CommunityK624/jev-localization-bench
CodeCut article by Khuyen Tran comparing TypeSafe Jev, RapidFuzz, and Sentence Transformers on the Abt-Buy product matching benchmark, with per-tool merge counts and a walkthrough of asking Jev which product is the same.
Jev Benchmarks & EvaluationsArticle#evaluation#benchmark#classification
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.