TypeSafe Jev Calibrate For Code Review

Only when they share context. Over 82 comment blocks, one request per comment produced 25 false flags, batching about 16 comments per request produced 11, and adding the whole file to each request produced zero flags at 799k tokens. Batching without shared file context does not just change the numbers: results reorder when ranked by confidence, so threshold tweaks only trade false positives for false negatives.

Both rule sets over four project files cost about 1.7M input tokens, roughly seven cents, at around 600 ms per request, with no slowdown observed at 48 parallel requests. Cheap in money, not in tokens: the shape that produced zero false flags was one request per comment plus the entire file included in each request, which is why the cheapest-looking option is also the noisiest one.

Not necessarily, because the threshold is a result of your codebase, your rule wording, and how measurable the rule is in the first place. The author started at 0.6 and left many rules there, and reports cases with a positive near 0.5 against a negative near 0.3, ordered correctly and consistently. Variance is also worst mid-range: one rule held 0.980 with zero spread while another wandered 0.53 to 0.65.

One person's notes on one Unity codebase, using the author's own thresholds, not a public benchmark suite, and the author calls it a work in progress he probably will not update. He also leaves the failures in: a verb_disagreement rule scores the class it fixed worse than the original regardless of how much context you add, and a verb_overload rule only worked while its example closely resembled the real code.

It changes them far more than adding questions, which barely moves the numbers. In one table the same comment and code pair scored 0.32 with a single referenced pair, but 0.75 when fourteen unreferenced pairs were also in the state. The takeaway is not to batch topics that do not share context, since thresholds and ranking do not carry over, and to add context only when a rule needs it.

It started at 0.6 and many rules stayed there rather than switching to the common 0.8. It found cases where a positive sat near 0.5 and a negative near 0.3 with the scores ordered correctly and consistently, suggesting a usable threshold can exist below 0.5. The right value depends on the codebase, the rule wording, and how measurable the rule is.

Twelve runs of the same cases showed variance concentrated mid-range: a flag-real case had a 0.53 to 0.65 spread while a pass control stayed 0.14 to 0.16 and a confident flag sat at 0.98. Confident answers are solid and uncertain ones wobble, which the author reads as the reason higher thresholds get recommended.