Code review is already a decision pipeline. Someone decides which changes are risky, who should read them, which comments are worth posting, and whether the merge gate opens. Most teams run that pipeline on intuition and a diff view. A decision model can make the pipeline explicit, but it cannot make the review true.
The four decisions in a review
| Decision | Question | Typical output | What a model can add |
|---|---|---|---|
| Risk tier | How dangerous is this change? | P0, P1, P2 | A bounded classification with confidence |
| Reviewer routing | Who should look at it? | Team or person | A choice from a fixed roster |
| Comment triage | Is this finding worth posting? | Flag or pass | A score or probability per rule |
| Merge gate | May this land? | Allow, ask, block | A signal, not the authority |
A model can rank a diff, route it, or flag a hunk. It cannot prove a defect, and it should not be the only reason a merge gate opens.
Diff risk is not diff size
The Hacker News discussion captured in Jev-based code review on Hacker News is a useful reality check. The thread drew 47 points and 56 comments on 25 September 2026. The author's example is an agent pull request with 230 changed files that people merge unread; the tool ranks each change P0, P1, or P2 and shows only P0 by default. The catalog is careful to note that the 230-file figure is the author's example of the problem, not a measured review of that diff, and that no detection rates or false-negative counts are published.
Commenters pushed back on the framing rather than the classifier: if agents routinely produce 230-file pull requests, the process is the problem, and the same agent could split them. One called Jev a generic classifier like the random forests and logistic regressions of 15 years ago. That is a fair question for any review tool.
Jev Review is more explicit about its limits. It reviews a diff or scans a codebase, printing JSON or saving reports for a local dashboard. It screens correctness, security, reliability, compatibility, and test coverage. The orchestration stays in code: a Noul risk matrix, Choice and Score file profiles, Choice evidence selection, Choice mechanism classification, Score severity, and a conditional Choice routing step. The README says it has no compiler diagnostics, static analyzers, repository indexing, or generated explanations; findings are review prompts, not proof of a defect. It needs Node 24 or newer, Git, and a TypeSafe key, and publishes no call count or cost estimate.
A review prompt is a place to start reading.
Context changes the answer
The most useful calibration write-up in the catalog is typesafe-jev-calibrate-for-code-review. Over 82 comment blocks, one request per comment produced 25 false flags; batching about 16 comments per request produced 11; adding the whole file to each request produced zero flags at 799k tokens. Batching without shared file context does not just change the numbers; results reorder when ranked by confidence, so a threshold tweak only trades false positives for false negatives.
The cost is manageable: both rule sets over four files took 1.7M input tokens, roughly seven cents, at around 600 ms per request, with no slowdown at 48 parallel requests. The threshold was not the usual 0.8; the author started at 0.6 and left many rules there, finding cases with a positive near 0.5 and a negative near 0.3, ordered consistently. Variance was worst mid-range: one rule held 0.980 while another wandered from 0.53 to 0.65. The study is one Unity codebase and the author's thresholds; the repository has no license file, and the author calls it a work in progress. The failures stay in: a verb_disagreement rule scores the class it fixed worse than the original, and verb_overload only worked while its example resembled the real code.
Batch only comments that share context, add the file when a rule needs it, and recalibrate rather than copying 0.8.
Reviewer routing and labels
Reviewer routing is a Choice question with a fixed roster, and the same discipline applies to labels. jev-pr-labeler assigns type, area, platform, and conceptual-scope size from a fixed taxonomy. Size is never line counts: XS is a trivial correction, S a focused change within one component, M a substantive subsystem feature, L cross-subsystem behavior or an interface change, and XL an architectural redesign or migration. A small protocol change can carry broader scope than a large mechanical rename.
The safety rails are worth copying. Unknown or low-confidence answers below 0.75 abstain, and manual labels always win. Dry-run is the default and never writes to GitHub, though it still makes a paid call. Evidence is bounded to 2 MB and 64 excerpts; comparisons over 300 files are rejected. The hosted workflow waits for a completed Greptile review of the current head before labeling. Title, description, filenames, and patches go to OpenRouter/TypeSafe.
Composite scores keep policy visible
When a review needs several dimensions, do not ask one question to produce a verdict. Composite scoring breaks the judgment into atomic Score questions, normalizes each by its top level, and combines them with weights in code. The resume example asks four Score questions in one parallel request, each with five levels, normalizes each score by dividing by len(criteria) - 1, then applies role-specific weights: 40% Python, 10% leadership, 40% design, 10% generalist for a senior IC; 15% Python, 40% leadership, 20% design, 25% generalist for an engineering manager.
The weights are policy, not model output; if the results do not match expectations, adjust the weights rather than re-prompting.
Calibrate thresholds locally
Confidence is a 0–1 statistic derived from the probability distribution; Noul answers carry no confidence at all. The docs' example routes anything below 0.5 confidence to a human and requires above 0.9 to auto-approve a transfer, while a read-only lookup is not gated as hard. The values depend on your domain and the model's performance on your use case, and the docs advise starting conservative and tuning on your own data.
For review, that means a threshold per action: a comment, a request-for-changes, and an auto-merge should not share a floor. Jev QA Analyzer shows the two-question pattern on 70 synthetic QA tickets: a 10-level rubric scores release risk against the dataset's reference score, while a 9-label Choice diagnoses the main risk cause and is explicitly exploratory because there is no ground truth. The scripts exclude qa_owner, risk_score_10, and release_readiness from the state. The README is truncated, the data is synthetic, and no license is listed, so the demo is a shape, not a result.
What not to automate
- Do not treat a finding as proof. A model can point at a hunk; it cannot compile the change.
- Do not hide P1 and P2 without an owner for the hidden tier.
- Do not batch comments that lack shared context; the ranking will move.
- Do not copy a 0.8 threshold from another codebase.
- Keep a human review path for the cases the model marks uncertain, and log the decision so the next calibration has data.
jev-guard is a useful model for the gate itself: ALLOW, ASK, or DENY, with a fast local path, a semantic path for ambiguous calls, a fail-safe default that asks when the API is unreachable, and audit mode for trying the policy before enforcing it.
For the broader integration, see Jev in Coding Agents; for threshold design, Confidence Gates in Production; and for the failure catalogue, Designing Around Jagged Edges. A confidence score is an input to a policy, and composite scoring is how you keep that policy visible.
