What does this benchmark compare?

Dozens of zero-shot systems across extractor encoders, classifiers, rerankers, and typed-decision engines, including the Jev family and its open lookalikes, scored on measured accuracy, latency, and cost in one repository.

Are the systems local or hosted?

Both: models can run on your machine, ten of them are served by the local Ollaya daemon, and hosted ones sit behind OpenRouter's decision and rerank endpoints plus Fastino's API and TypeSafe.

What is Ollaya in this context?

A local daemon that serves ten of the benchmarked models from one process, so systems like Laya, von, Kev, decider, and the Decision family can be compared without per-call cost.

How are decision models like Jev scored here?

They implement the System One-style typed-decision contract, choice, score, and noul, and are benchmarked on the same jobs and metrics as the extractor encoders, classifiers, and rerankers.

Is this a Jev benchmark or an NLP benchmark?

Both: information extraction and classification are the jobs, and typed-decision engines are one of the families compared, alongside GLiNER-style encoders, rerankers, and NLI models.

What does the web UI offer?

A Gradio web UI that demos the systems and presents the benchmark charts; the README also lets you skip straight to the charts if you only want results.

Which license applies to the models?

The repository is MIT, but each benchmarked model keeps its own license, from Apache 2.0 to CC BY-NC-SA 4.0 and the LFM Open License, so check the specific system before you reuse it.

CommunityJev Benchmarks & Evaluations

Zero-Shot IE Bench

A benchmark harness and web UI that cross-compares dozens of zero-shot information-extraction and classification systems — extractor encoders, rerankers, and typed-decision engines like Jev — on accuracy, latency, and cost.

See highlights

Open github.com

Stars: 7
Language: Python
License: MIT
Published by: Community
Author: umstek
Last verified: Oct 2026

umstek/zero-shot-ie-bench

User repository on GitHub

View on GitHub

60 zero-shot information-extraction & classification systems (57 benchmarked) across 32 families — extractor encoders, zero-shot classifiers, cross-encoder rerankers, typed-decision engines — local, Ollaya-served & hosted (OpenRouter, Cloudflare, TypeSafe); measured accuracy/latency/cost benchmarks, 9-language suite, web UI

Forks: 2
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.

Tagscommunity python benchmark information-extraction gradio

Works forJev for Extraction Jev for Classification Jev for Evaluation

Used inJev for Software & DevTools Jev for Data & Analytics

Languages: Python
Platforms: OpenRouter Cloudflare
Capabilities: Open sourceRuns locallyRequires API key

Highlights

Cross-compares local, Ollaya-served, and hosted systems on measured accuracy, latency, and cost.
Covers extractor encoders, zero-shot classifiers, cross-encoder rerankers, and typed-decision engines.
Typed-decision engines include Laya, von-1.0, Kev, AgentJev, decider, and MoJev on the System One-style contract.
Ten models are served by the local Ollaya daemon; hosted ones sit behind OpenRouter, Fastino, Cloudflare, and TypeSafe.
The README reports 76 zero-shot systems, 73 of them benchmarked, across 35 IE and classification families.

Watch out

MIT for the repository, but each benchmarked model keeps its own license, from Apache 2.0 through CC BY-NC-SA 4.0 and the LFM Open License, so check per system before reuse. Hosted systems need provider keys, and some weights are gated on Hugging Face.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

System One on Snapdragon

A dated harness for running open System One typed-decision models — Laya, Decider, imajev, Winnow-12B, decider-12b — on a Snapdragon X Elite NPU, GPU, and CPU, benchmarked against TypeSafe's Jev API.

Jev Benchmarks & Evaluations#community#python#snapdragon

Community

esterhuizen/system-one-on-snapdragonPython

What the Jev

A reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.

Jev Benchmarks & Evaluations#community#python#evals

Community

keta1930/what-the-jevPython

hev reranker

A BEIR reranking recipe that asks up to 30 parallel Noul relevance questions in one Jev state, with RESULTS.md against cross-encoders and hosted rerankers.

Jev Benchmarks & Evaluations#community#evaluation#python

Community

hev/rerankerPython

jev vs. open alternatives

Notebook benchmark that compares Jev with open models and specialised tools on five document-pipeline chores: language detection (Choice(12)), page orientation (Choice(4)), RVL-CDIP classification (Choice(16)), bundle splitting and parse-tier triage, with liteparse doing all PDF rendering and OCR work.

Jev Benchmarks & Evaluations#community#evaluation#document-processing

Community

run-llama/jev_vs_ossJupyter Notebook

Open-Jev v1.1 Data

326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.

Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation

Community

JevEmbed Data

1,601,157 training and 66,482 test decision questions for JevEmbed, covering choice, score and noul types across ten Parquet shards of at most 200,000 rows each.

Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation

Community

Back to all resources

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai