Jev directory
CommunityJev Benchmarks & Evaluations
Think Fast Jev Queries
Hugging Face dataset of 150 crypto-domain user queries, each labeled with a router intent and a guardrail decision. Rows carry id, text, router, guardrail, and source fields, covering 14 router labels and allow, advice, and off_topic guardrails.
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- 150 rows in a single test split, stored as JSON and auto-converted to Parquet.
- Each row has id, text, router, guardrail, and source fields.
- Router labels include price, market_chart, derivatives, onchain_dex, tickers, nfts, global, and treasury.
- Guardrail values are allow, advice, and off_topic; source values are seed and generated.
- Listed as a text classification task in English under a cc-by-4.0 license.
Questions about this entry
Short answers drawn from the linked project or page.
It is a labeled set of 150 crypto-domain user queries, each paired with a router intent label and a guardrail label. It is listed on Hugging Face as a text classification dataset and can be used to check how a router and guardrail layer classify real-looking questions.
The dataset has 150 rows total, held in a single split named test. The size is listed as under 1K, and the viewer shows one default subset with those 150 rows.
Router values include price, market_chart, derivatives, onchain_dex, tickers, coin_details, coin_markets, trending, categories, exchanges, nfts, global, treasury, and unsupported. The unsupported label covers questions the router is not meant to answer.
The guardrail column takes the values allow, advice, and off_topic. Rows such as investment-suggestion questions are labeled advice, and unrelated questions such as cake recipes are labeled off_topic, while ordinary data lookups are labeled allow.
The dataset card lists a cc-by-4.0 license, which permits reuse with attribution. Check the card on Hugging Face for the current terms before redistributing the rows.
The card lists the Datasets library along with pandas and Polars, and the files are distributed as JSON with an auto-converted Parquet version. That means it can be pulled with the Hugging Face Datasets loader or read directly as JSON or Parquet.
It is small at 150 rows, English-only, and scoped to crypto questions, with rows marked as either seed or generated. It covers routing and guardrail labels only, so it does not measure answer quality or calibration on its own.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Facebook: Intake Router keeps diagnosis out of the answer space at the front desk.
FacebookClinicnotes
X: OpenRouter Decision Model Rankings
X: herdr + jev, an agent router
X: A group chat bot with decision room routing
More Jev resources like this
Independent experiment scoring 400 real job postings against 30 fictional résumés with Jev, producing 12,000 evaluations in about 197 seconds for roughly $0.17, with a 100-posting answer key and an analysis of what the model got wrong.
Jev Benchmarks & Evaluations#github#madewithjev#evaluation
Communitykaucap08/jev-job-matching
Controlled benchmark of Jev against Claude Sonnet 5 on one coding-agent decision: given a bug report, which file needs editing? It scores 200 real SWE-bench Verified issues with cost, latency, and confidence-band routing tables.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
CommunityK624/jev-localization-bench
Spring Boot and Spring AI benchmark comparing Java keyword rules, Jev 1.13.0, and Claude Haiku 5.5 on warehouse goods-receipt triage, with a 200-remark synthetic dataset and a Python analysis script.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
Communitymohammedahmadi1990/jev-triage
arXiv paper presenting the first systematic bias and fairness audit of Jev, a typed-decision model for recruitment. It measures how demographic attributes and candidate cues shift screening scores, rankings, and shortlisting, against a noise floor, plus validity, latency, and cost.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#research
Community
arXiv paper presenting Slot Attack, a black-box attack that appends text to evidence to steer decision models toward an attacker-chosen wrong option. Evaluates seven decision models on 16 Decision Bench tasks across seven domains.
Jev Benchmarks & EvaluationsArticle#arxiv#security#evaluation
Community
arXiv evaluation of 14 System-1 decision models, including Jev, against four generative judges on eleven judge benchmarks, covering accuracy, calibration, cost, and robustness to instructions injected into judged responses.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.