Organization repository on GitHub
- Forks
- 2
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
CommunityJev Benchmarks & Evaluations
Jev Vs. Open Alternatives
Notebook benchmark that compares Jev with open models and specialised tools on five document-pipeline chores: language detection (Choice(12)), page orientation (Choice(4)), RVL-CDIP classification (Choice(16)), bundle splitting and parse-tier triage, with liteparse doing all PDF rendering and OCR work.
- Stars
- 7
- Language
- Jupyter Notebook
- License
- MIT
- Published by
- Community
- Last verified
- Oct 2026
About this project
Material comparing jev (TypeSafe AI's hosted decision model) with open alternatives on four document-pipeline chores, with liteparse (pdfium + tesseract) doing all the PDF work. build-corpus --n N rebuilds one corpus at a different size. Or, just read the notebook to follow along. Small "binaries" over jevvs/pdftools.py, installed by uv sync: jev docs: — models & pricing, confidence, jaggedness.
Quoted from the project's README. Read the full README.
Used inJev for Data & Analytics
Highlights
- Five chores asked as Choice(2) up to Choice(16) over generated, labelled corpora.
- Compares Jev against Qwen3.5-4B on Modal, local Laya and jeff encoders, and lingua/tesseract OSD.
- build-corpus all downloads, renders and OCRs PDFs locally in about 5-10 minutes.
- Ships liteparse CLIs: pdf-info, pdf-text, pdf-render and pdf-orient.
Watch out
MIT licensed. The repository has no GitHub description or topics and this is a notebook-scale evaluation, so results depend on generated corpora rather than a published leaderboard; the Modal comparison row requires a Modal account.
Questions about this entry
Short answers drawn from the linked project or page.
Open decoder Qwen3.5-4B served on Modal with SemIf-style direct logits, the local open encoders Laya and jeff, and specialised tools lingua for language detection plus tesseract OSD for orientation, across five document chores all rendered through liteparse.
The Jev rows need a TYPESAFE_API_KEY, the Modal rows need a Modal account, and local encoders run on your own hardware. Corpus building and the liteparse PDF helpers run locally after uv sync without any provider calls.
Language detection as Choice(12) over Wikipedia excerpts rendered to PDF, orientation detection as Choice(4) on 0/90/180/270-degree rotated pages, RVL-CDIP document classification as Choice(16), bundle splitting as Choice(2) per page boundary on concatenated multi-page articles, and parse-tier triage as Choice(2) on liteparse complexity signals plus an OCR excerpt, labelled by OCR character error rate against the source.
It does all the PDF rendering and OCR work in the comparison: pdfium plus tesseract. Four small CLIs sit on top of jev_vs/pdf_tools.py: pdf-info reports pages, producer and which pages need OCR and why; pdf-text gives per-page text, OCRing only where there is no native text; pdf-render rasterises pages, optionally rotating into a fake scan; and pdf-orient compares OCR at four rotations against tesseract OSD.
Laya is a 421M ModernBERT with a decision head speaking the jev-compatible state and questions schema, run locally on MPS or CPU using the typed-decisions checkpoint with max_len raised to 2048 and head_max_len to 512. jeff is a self-hosted server on gliformer-large-v1 at 400M that speaks the same wire format, so the notebook drives it through a JevBackend and just a base_url.
The repository has no GitHub description or topics and this is a notebook-scale evaluation rather than a published leaderboard, so the rows depend on generated, labelled corpora rather than a fixed benchmark. It also carries no license beyond MIT, and the Modal comparison row requires a Modal account to reproduce.
uv sync, copy .env.example to .env with a TYPESAFE_API_KEY, then uv run build-corpus all, which downloads the samples, renders and OCRs the PDFs into ./data in about 5 to 10 minutes. build-corpus <task> --n N rebuilds one corpus at a different size. Then uv run modal deploy modal_app.py serves Qwen/Qwen3.5-4B on an L4, and uv run jupyter lab opens jev_vs_open.ipynb.
More Jev resources like this
8.3kGitHub stars
Agentic search evaluation inside Zilliz DeepSearcher: a standalone harness replaces the ChainOfRAG reflection call with one Jev Noul per round to decide whether to stop within a seven-round budget, then compares stopping behavior, evidence recall and decision cost on 100 sampled 2WikiMultiHopQA queries.
Jev Benchmarks & Evaluations#community#rag#evaluation
Community
zilliztech/deep-searcherPython
zilliztech/deep-searcherPython4.9kGitHub stars
LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.
Jev Benchmarks & Evaluations#community#typescript#evaluation
Community
langwatch/langwatchTypeScript
langwatch/langwatchTypeScript15GitHub stars
A BEIR reranking recipe that asks up to 30 parallel Noul relevance questions in one Jev state, with RESULTS.md against cross-encoders and hosted rerankers.
Jev Benchmarks & Evaluations#community#evaluation#python
Community
hev/rerankerPython
hev/rerankerPython3GitHub stars
An open benchmark runtime for document-grounded decision models: it scores Jev Noul, Choice and Score answers on pinned tasks and counts malformed probability distributions as misses so model comparisons stay reproducible.
Jev Benchmarks & Evaluations#community#benchmark#evaluation
Community
Hanno-Labs/decision-benchPython
Hanno-Labs/decision-benchPython0GitHub stars
Independent pre-registered audit of Jev on Spanish: 19,200 calls over 3,200 paired items on XNLI, PAWS-X, MASSIVE and Belebele. A Spanish state costs 3.0-6.4 accuracy points and roughly doubles ECE on the two hardest tasks, while Spanish instructions change nothing; a CLI reruns the same comparison on your own data.
Jev Benchmarks & Evaluations#community#evaluation#calibration
Community
marcosmartinez/jev-acentoPython
marcosmartinez/jev-acentoPython6GitHub stars
Chinese-language research project for multimodal social-risk moderation: a local Qwen3-VL-32B with language-side LoRA emits structured evidence JSON, and a fixed Jev policy turns that evidence into risk bands, with frozen test leaderboards and pure-Jev ablations.
Jev Benchmarks & Evaluations#community#moderation#multimodal
From the Jev community
Posts from builders shipping with Jev right now.