Docs
CommunityJev Benchmarks & Evaluations
SocialJev Evals
An evaluation set from the social-jev account: 2,597 validation and 2,553 test rows carrying dataset, document and state ids, a query type, a role/content message list and an integer target. The repository carries no prose card, so the task, label meaning and licence are undocumented.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- SocialJev
- Last verified
- Oct 2026
Highlights
- Two splits: 2,597 validation and 2,553 test examples, about 8.4 MB of parquet data in total.
- Columns: id, dataset, document_id, state_id, query_type, messages (role/content list) and an int64 target.
- messages is a list of role and content strings; dataset and query_type are per-row strings.
- Published under the social-jev account, created 2026-10-13; the card contains only the dataset_info block.
Watch out
Card reviewed 2026-10-17; the repository carries no prose card, so the task semantics, label meaning, provenance and licence are not documented anywhere on it. Everything above comes from the dataset_info metadata that the digest captured, and no usage example, benchmark or field documentation is available on the card.
Questions about this entry
Short answers drawn from the linked project or page.
A small evaluation dataset published under the social-jev account, with 2,597 validation and 2,553 test examples in parquet. Each row pairs metadata about a document and a query with a chat-style message list and a single integer target.
Eight features: id, dataset, document_id, state_id and query_type as strings, messages as a list of role/content string pairs, and target as an int64. The metadata suggests scoring a query over a state or document, but the card defines no field.
The validation split holds 2,597 examples (about 4.2 MB) and the test split 2,553 (about 4.2 MB), for a dataset size near 8.4 MB against a 7.9 MB download. There is no train split and no config beyond the default.
None is stated. The card contains no licence field, no licence tag and no prose, so the terms of use are undocumented and reuse rights cannot be determined from the repository alone.
It is an ordinary parquet dataset with a default config, so the datasets library can load it directly; the card itself gives no usage example, column documentation or field explanations beyond the dataset_info metadata captured here.
The column names - state_id, query_type, target - resemble decision-model evaluation records, but the card never mentions Jev, TypeSafe or System One, so no connection to the typed-decision contract can be confirmed from this repository.
The social-jev account, created 2026-10-13. Everything above is from the dataset_info block; the meaning of query_type and target, the provenance of documents and messages, and the intended metric are all unstated.
More Jev resources like this
LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.
Jev Benchmarks & Evaluations#community#typescript#evaluation
Community
langwatch/langwatchTypeScript
langwatch/langwatchTypeScriptA 770-post benchmark that scores Jev, Sonnet 5, GPT-5 nano, and two local models on Reddit verdicts with weighted Brier scores.
Jev Benchmarks & Evaluations#community#python#evaluation
Community
dchristopoulos/jev-aitaPython
dchristopoulos/jev-aitaPythonTeacher-labelled distillation pack for the one-token Qwen decision track: 10,044 rows across train, eval and calibration JSONL files, plus 1,708 labelled fixtures. Labels come from jev-1.13.0, and the card says not to train on the eval or calib splits. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#distillation
Community
The research record of the Xyntetik typed-decisions track: TD-000 measurements and the preregistered TD-001 and TD-002 tests of whether /v1/decide probabilities from Xyntetik-Kvist-14B are usable as decisions. Both tests are INCONCLUSIVE; the record holds questions, scored rows, analyses and the harness. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
xiq's JEV-format view of OpenJev-Data: every browser step re-expressed as state, questions and gold across eight configs covering Mind2Web, NNetNav, WebChain, two teacher runs and on-policy GLM, plus a frozen 2,346-step bench and 153 live tasks. Licences are mixed and per-row terms apply.
Jev Benchmarks & EvaluationsDocs#community#evaluation#browser-agent
Community
johannhartmann's German translation of the LocalLLaMA/typed-decisions benchmark at revision c806f6c: states, instructions, options and rubrics translated to German while keys, ids and gold distributions stay as in the original. Machine-translated without human review. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#typed-decisions
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.