User repository on GitHub
A small second eval for shadcn-ui/lint that uses TypeSafe's Jev to judge the linter's own output.
- Forks
- 0
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
CommunityJev Practices & Patterns
Jev Shadcn Lint Eval
An eval that puts Jev two questions about shadcn-ui/lint: whether each finding is a real rule violation, and whether the linter's message tells you what to change. It runs the real linter over 131 rule test cases and scores both the detections and the messages.
- Stars
- 0
- Language
- JavaScript
- Published by
- Community
- Author
- blas0
- Last verified
- Oct 2026
About this project
This checks shadcn-ui/lint (commit 53de86f) with TypeSafe's Jev model (jev-latest). The repo's own eval measures how agents style UI before and after lint feedback. This one asks Jev whether each lint result is right and whether its message tells you what to change. design-system.json: the source of truth for what Jev is told about each project: the real color tokens, component files and their variants for five shadcn-ui/lint fixtures. extract-design-system.mjs regenerates it.
Quoted from the project's README. Read the full README.
- Languages
- JavaScript
Highlights
- 131 rule test cases, 79 that should be reported and 52 that should not, executed through the real linter with every answer saved.
- 82% of cases answered correctly at a 0.7 probability threshold; 91% of real violations caught, but 16 of 52 clean cases flagged.
- The linter's 115 messages scored 0 to 2 for actionability, averaging 1.41; 'its contract denies flex' was among the weakest.
- design-system.json carries real color tokens and component variants for five shadcn-ui/lint fixtures, regenerated by a script.
Watch out
The accuracy figures come from a single recorded run: 246 requests with no errors and 249,384 input tokens, and Jev bills input tokens only. Correctness counts a probability of 0.7 or more as yes, so the 82% depends on that threshold. Most false positives are cases using something a rule deliberately ignores, such as bg-[#333] under no-raw-colors, because rules.json does not yet tell Jev what each rule ignores. Adding component data did not clearly change accuracy: two earlier runs without it scored 79% and 82%. Running this needs a local checkout of shadcn-ui/lint with pnpm install and pnpm build, and rebuilding the case file means temporarily rewriting createTester() in that repo's test helpers. The agent-run path is unfinished: build-states.mjs was tested once on a made-up run and has not been sent to Jev. Copy reviewed against the repository README on 2026-10-04, not from running it.
Questions about this entry
Short answers drawn from the linked project or page.
One real run against the lint repo's actual components and theme colors: 246 requests with no errors and 249,384 input tokens, with every answer written to records.json and the scores to summary.json under results/. Correctness counts a probability of 0.7 or more as yes. You need a checkout of shadcn-ui/lint with pnpm install and pnpm build done, an exported TYPESAFE_API_KEY that is never committed, and node run-rule-cases.mjs with an optional --limit n.
Jev caught 91% of real violations but flagged 16 of the 52 clean cases, so it is noticeably weaker on code that should not trip a rule. The README attributes most of those misses to constructs a rule deliberately ignores, such as bg-[#333] under no-raw-colors, which rules.json does not describe to the model yet. Telling Jev what each rule ignores and rerunning is the first listed next step, so treat 82% as improvable rather than settled.
Not clearly. Two earlier runs without component data scored 79% and 82%, against 82% for the run that included it, so the design-system extraction did not move accuracy. That file holds the real color tokens and component files and their variants for five shadcn-ui/lint fixtures, and extract-design-system.mjs only needs rerunning when the lint repo itself changes.
No, not yet. The upstream lint repo has its own eval for that; this project asks Jev whether each lint result is right and whether its message says what to change. The script that turns an agent run into states for Jev, build-states.mjs, was tested once on a made-up run and has not been sent to Jev, and comparing its answers against the lint repo's own outcome labels is still an open next step.
design-system.json holds real color tokens and component variants for five shadcn-ui/lint fixtures, rules.json lists the six rules and how to fix each, rule-cases.jsonl has 131 cases (79 that should be reported and 52 that should not), and run-rule-cases.mjs runs the real linter, asks Jev and writes records.json plus summary.json under results/.
You need a checkout of shadcn-ui/lint with pnpm install and pnpm build done, then export TYPESAFE_API_KEY and run node run-rule-cases.mjs rule-cases.jsonl --repo <lint checkout> with an optional --limit n. Regenerating design-system.json or rule-cases.jsonl is only needed when the lint repo changes, and rebuilding the case file means temporarily editing createTester() in its test helpers.
Jev scored the linter's 115 messages from 0 to 2 for whether they say what to change, and they averaged 1.41. The weakest, such as its contract denies flex, say what is wrong but not what to use instead.
More Jev resources like this
24GitHub stars
Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
rorshopping/jev-on-a-laptopPython
rorshopping/jev-on-a-laptopPython23GitHub stars
Live race rig that labels 1,000 real app reviews with Jev typed questions against Gemini JSON output on identical batches, then replays recordings and re-ranks rows locally without extra model calls.
Jev Practices & Patterns#community#jevbest#catalog-import
16GitHub stars
Vendor-neutral harness that tests whether Jev reranking improves a small RAG stack, using frozen hybrid BM25 plus BGE-M3 candidates and isolating retrieval, rerank, and generation stages.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
erendikmenn/jev-rag-benchmarkPython
erendikmenn/jev-rag-benchmarkPython7GitHub stars
Benchmark harness that tests whether models can forecast federal motion-to-dismiss outcomes from the written record, scored per claim-defendant pair with micro-Brier and clustered intervals.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
johnhughes3/LegalForecastBenchPython
johnhughes3/LegalForecastBenchPython6GitHub stars
Frozen 100-question-per-cell check of TypeSafe Jev on Korean versus English reading, paraphrase, and medical exam items, with recorded responses and uncertainty intervals.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
mahlernim/jev-korean-benchmarkPython
mahlernim/jev-korean-benchmarkPython7GitHub stars
Word-level language model experiment that uses Jev as its output layer, with explicit tokenizer, sampler, repetition mask, stop rules, and chunk verification.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
y0usaf/jev-lmTypeScript
y0usaf/jev-lmTypeScriptFrom the Jev guides
Original write-ups on what this entry is for.
Most Jev accuracy problems are state problems. What belongs in a state, what is a pre-made judgment to leave out, and why naming the slice you ask about matters more than the wording.
Jev Practices & Patterns#state#design#primitives
Read article
A Noul returns one number and no separate confidence. How to read the value, pin its boundary with criteria, split compound questions, and set a per-action threshold that matches the cost of being wrong.
Jev Practices & Patterns#noul#primitives#confidence
Read article
A Score is a weighted position, not a ruler. Why the number can land between levels, why levels are judged separately, and how to combine several atomic scores in code instead of one crowded rubric.
Jev Practices & Patterns#score#primitives#scoring
Read article
From the Jev community
Posts from builders shipping with Jev right now.