Question 1

Can it tell a real citation from a fabricated one?

Accepted Answer

No, and the README says so as a limit rather than burying it. On a made-up source it returned 0.80 against 0.87 for the real one, so the tool is meant for reproducible verdicts on criteria, not for fact-checking, citation or accountability work. Use it to rank or gate candidates, then check the sources yourself.

Question 2

Why four questions per judgment instead of just one?

Accepted Answer

Because they tried one and it failed in a specific, measurable way: pseudo-scientific nonsense scored 0.74 to 0.82 on a single question where a real textbook passage scored 0.48, so one question shifts the assessment rather than measuring it. The skill ships a four-question rubric instead, and tells you to calibrate the threshold on a contrast pair before trusting any number.

Question 3

What does a judgment actually cost compared with asking an LLM?

Accepted Answer

The README puts a call at roughly 306 input and 20 output tokens and about two seconds, against thousands of tokens and tens of seconds for a normal LLM, and reports run-to-run spread of 0.004 to 0.014 over eight repeats. It also notes that a plain model tends to sound confident where Jev returns 0.5.

Question 4

What happens when the score lands in the middle?

Accepted Answer

The skill treats 0.30 to 0.70 as uncertain and sends it to a human, calling any attempt to squeeze a verdict out of that band the main mistake in working with the tool. A lukewarm answer is meant to be read as the model not knowing, so it is a result rather than a failure to retry.

Question 5

What does the paragraph scanner actually map?

Accepted Answer

scripts/scan_text.py walks a document paragraph by paragraph and marks where the text argues with grounds, lists, asks a question, or admits uncertainty. Run it as python3 scripts/scan_text.py article.md --mode argument --min-len 300 and it prints a paragraph map plus JSON next to the file. The measured bounds: pseudo-scientific nonsense over 5 paragraphs scored a mean of 0.362 with 0 checkable, 4 needing a human and 1 general words, while a 116-paragraph analysis textbook scored 0.690 with 52 checkable and 64 needing a human.

Question 6

How sensitive is a score to how the question is worded?

Accepted Answer

Enough that the README measures it. Replacing принципиально отличается with a weaker formulation raised the conclusion-following score from 0.504 to 0.546, with a within-case spread of 0.009 to 0.014, and the difference is reported as significant. That is why it insists on calibrating the threshold against a contrast pair first. Calibration testing covered 8 known-answer cases with 6 of them inside 0.15, and symmetric pairs landed at 0.99 against 0.01 and 0.98 against 0.02.

Question 7

How do I install it, and what does the wrapper expose?

Accepted Answer

It is a Letta agent skill: letta install the URL of skills/judging-with-typesafe, and set TYPESAFE_API_KEY from console.typesafe.ai/keys. scripts/typesafe.py wraps the API with noul for the probability of yes, choice for a selection from a list, score for a position on a scale, and batch for many questions over one state. A worked example asks python3 scripts/typesafe.py noul "В корзине 12 яблок и 5 груш." "Яблок больше, чем груш?" and returns 0.99. The README is in Russian.

Judging With TypeSafe

Highlights

Questions about this entry

More Jev resources like this

From the Jev community