Jev directory
CommunityJev Benchmarks & Evaluations
Jev History Corpus: Post-Cutoff Knowledge Canary
A synthetic question-answer corpus about the Jev family of decision models, published as a diagnostic for fine-tuning pipelines. Version 1.0 ships 10,389 training pairs and 265 held-out evaluation questions drawn from 58 source units, so a training run can be scored before and after.
- Reading time
- 3 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Version 1.0 contains 10,389 training pairs and 265 held-out evaluation questions from 58 source units.
- Built from 2,616 extracted fact records; every training fact is covered by at least one pair.
- Question types include numbers, comparison, definition, limitation, mechanism, recipe, why, timeline, and unknown.
- Released under apache-2.0 with train, meta, and eval subsets loadable through load_dataset.
- Subject matter postdates September 2026, so a model with an earlier pretraining cutoff should score near zero before training.
Questions about this entry
Short answers drawn from the linked project or page.
It is a diagnostic for fine-tuning pipelines rather than a knowledge resource. Train on the corpus, score the held-out questions before and after, and the difference shows whether a training run is writing new knowledge into the weights. Typical uses include validating a new training method, comparing LoRA with full fine-tuning, checking a data loader or chat-template handling, and smoke-testing a knowledge-injection pipeline.
Version 1.0 has jev_qa_train.jsonl with 10,389 training pairs, jev_qa_train.meta.jsonl with 10,389 row-aligned provenance records, jev_heldout-v0-1.jsonl with 265 evaluation questions plus gold answers and grading keywords, and build_report.json with row counts, per-unit and per-type counts, and exclusions.
Use the datasets library with the train, meta, and eval subsets: load_dataset("juiceb0xc0de/jev-history-corpus", "train")["train"] for the pairs, the same call with "meta" for provenance, and "eval" for the held-out questions. Training rows are single-turn chat records with user and assistant messages.
The dataset card lists apache-2.0. It is hosted on Hugging Face under the user juiceb0xc0de, tagged for question answering and text generation, English language, text modality, and a size of 10K to 100K rows.
Grade against the gold answers, either by hand or with an LLM judge. Each evaluation row carries one to four keywords, but the card states those keywords are a screening aid for a quick first pass and are not sufficient on their own as an automated correctness check. There is one held-out question per held-out fact.
The Jev family has a well-defined start date. TypeSafe's launch post is dated September 15, 2026, the papers the corpus draws on carry arXiv identifiers from the 2609 and 2610 series, and the latest dated source is a usage guide from November 14, 2026. Coverage includes hosted Jev models, Open-Jev, the JEV-9B, JEV-27B, JEV-27B-VL and JEV-35B checkpoints, GEV-26B-Decide, and community derivatives such as NeoHorse-Jev-4B and Jeva-4B.
A date is not proof of absence. The Jev sources cite 2024 and 2025 papers, build on earlier base models such as Qwen3.5, use common benchmarks such as MMLU-Pro, and at least one source repository predates the September 2026 burst, so a model can earn partial credit by reasoning from general background. Measure the baseline on the eval set before training; the canary is informative only when the baseline is low and the post-training score is clearly higher.
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.