Skip to content
JevDirectory.org
Practices & Patterns#evaluation#state#dates#counting

Designing Around Jev's Jagged Edges

Jev cannot count, reads dates as text, and takes instructions literally. These are not bugs to work around quietly; they are design constraints with known guardrails.

Jev is not a small language model. It is a deliberately narrow decision model, and the fastest way to get value from it is to stop asking it to do things it is bad at. The official jaggedness list documents those limits. This article turns the list into design decisions.

Do not ask it to count

Counting is unreliable, and the error grows with the size of the set. The fix from the docs is mechanical: ask one question per item and count in code.

result = client.system_one(
    {"items": items},
    {
        f"item_{i}": Noul(instructions=f"Is `items[{i}]` the name of a fruit?")
        for i in range(len(items))
    },
)

count = sum(result.nouls[f"item_{i}"].noul > YES for i in range(len(items)))

The same logic applies to arithmetic, totals, and percentages. Jev labels; code computes. The pre-parsed value extraction cookbook applies the principle to money and identifiers: a regex finds candidate spans, Jev picks the requested one, and code parses the value with Decimal. Because Jev only chooses among spans that were found, the returned value is copied verbatim and cannot be invented.

Treat dates as text

Jev reads dates as strings, and text-based comparison loses. The date extraction cookbook solves it with seven Choice questions about the date's shape and parts (month, day, year, weekday) and resolves them in code against a pinned TODAY. next Thursday is ambiguous, so the code fixes a convention, and anything under 0.60 confidence goes to review. The absent-date example returned none at 0.46, which is a perfectly useful answer once a human sees it.

Say it literally

Literal reading is a feature when your instructions are precise and a failure when they rely on implied context. Two habits help:

  • Refer to state fields by key, in backticks, so the question points at something specific: `potential_duplicate`.
  • Spell out the comparison instead of naming it. "Does the claimed sender identity conflict with the sending domain" plus the two fields to compare beats "check for a spoof".

Advanced structure exists for exactly this: instructions, option descriptions, score levels, and criteria all accept JSON, so you can label the parts of a question and pass schemas or taxonomies directly.

Trim the state

Unrelated detail causes context rot: adding irrelevant material to the state degrades answers on the question you actually care about. The state guide recommends an object with descriptive field names over a wall of text, and the practical rule is to include what the question needs and nothing else. If a question can be answered from two fields, do not send the whole record.

Expect the invariants to break

Structural invariants do not hold. Complementary probabilities can sum past 1 (0.72 plus 0.47 in one documented case), and the same underlying question can disagree across primitives: a refund judgment read 0.22 as a Noul and 0.01 as a Choice. Do not build code that assumes P(x) + P(not x) = 1, and do not mix primitives for the same decision without checking they agree on your data.

Turn the list into tests

The jaggedness list is maintained but not exhaustive, so the durable move is to convert each failure mode into a fixture. Take a handful of labeled cases per failure mode, run them against your pinned model version, and assert on thresholds rather than exact values. The self-consistency cookbook is a good template for the harness, and the line-by-line search cookbook shows how to combine a positive question with an existence check so the system has a way to say the document does not contain an answer.

The mindset is the same throughout: use Jev for judgment, keep counting, arithmetic, dates, and final policy in code, and write down which side of that line each new feature falls on.

From the directory

The resources behind this article.

A maintained list of jev-1.13's known failure modes, literal reading, unreliable counting, dates as text, indirection, context rot, and contradictory criteria, each with a guardrail.
Practices & Patterns#official#docs#evaluation
Official
How to shape what Jev evaluates: a string, a named object, or an array of messages. Every question in a request sees the same state and is evaluated independently.
Practices & Patterns#official#docs#state
Official
Extract dates by asking seven Choice questions about a date's shape and parts, then resolving them in code: five of six examples auto-accepted and missing dates flagged at 0.46 confidence.
Cookbooks & Demos#official#cookbook#extraction
Official
Regex finds candidate emails, phone numbers, and amounts; Jev picks the requested span so code copies a verbatim value that cannot be invented, with no free-text extraction required.
Cookbooks & Demos#official#cookbook#extraction
Official
Semantic search over GitHub's Terms of Service: one request ranks all 218 lines with a Choice while a Noul checks whether the document contains an answer at all, including when it should say no.
Cookbooks & Demos#official#cookbook#search
Official
Back to all articles

More articles

The most common Jev design mistake is asking a question that does not match the primitive. Here is how to choose by the shape of the answer you need back.
Practices & Patterns#primitives#choice#score
Read article
Confidence is not accuracy, and a single global threshold is rarely the right policy. A practical guide to per-action gates, measured thresholds, and the logs you will want later.
Practices & Patterns#confidence#routing#evaluation
Read article

From the community

Posts from builders shipping with Jev right now.

Vercel's fx safety reviewer, 18x faster

We're seeing extraordinary results from @typesafeai. Default mode in 𝚏𝚡 is auto, with a safety reviewer analyzing every command. That reviewer runs on GPT Luna today. Jev is up to 18x faster (p95) *and* more accurate. It's coming to @vercel AI Gateway and likely new default.

Pranit
Pranit
Vercel
@fazxes

We benchmarked fx auto mode (safety) classifier with @typesafeai's Jev. tl;dr: ~5-18x faster and more accurate than 𝚐𝚙𝚝-𝟻.𝟼-𝚕𝚞𝚗𝚊, our current top choice

Image
Reply

Jev lands on OpenRouter