Skip to content
JevDirectory.org
Evaluation & Training

Contradictory criteria

When instructions and criteria ask for different things, accuracy drops; treat criteria as an extension of the instruction.

Category
Evaluation & Training
Also known as
—
Related terms
4
Directory entries
1
Docs
docs.typesafe.ai
Added
2026-09-24

Definition

The clearest example in the docs is a Noul whose true maps to no and false to yes, which performs worse than the natural phrasing. Clear, human-readable alignment between instruction and criteria is what the model was trained on.

The guardrail is to align the two with precise language and to prefer question shapes an average person could read and answer consistently.

Tagslimitationsdesign

From the directory

A maintained list of jev-1.13's known failure modes, literal reading, unreliable counting, dates as text, indirection, context rot, and contradictory criteria, each with a guardrail.
Practices & PatternsDocs#official#docs#evaluation
Official

From the community

Posts from builders shipping with Jev right now.

Follow @typesafeai

Computer use at 155× cheaper than Opus 5

i built computer use using @typesafeai ! it is 155x cheaper than opus 5, ~20x faster, and generalizes across OS's more on how it works in the vid & thread below:

Diogo Almeida
Diogo Almeida
TypeSafe AI
@CompleteSkeptic

After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x

Reply

Foreman keeps coding agents on task

Unclutter: an ad and slop blocker that runs on Jev

An open-source BS meter for debates and investor calls

🚨 Open Source Jev BS meter you can use this to analyze any debate / investor call / interview / sales pitch / podcast video fact check live , for example this dario interview cost 60 Jev calls / 111K tokens / $0.0047 github.com/ChetasLua/jevm…

Chetaslua
Chetaslua
@chetaslua

🚨 I gave the Trump vs Kamala debate a live BS meter using Jev every sentence, both candidates, 5 yes/no questions each 1,191 Jev calls / 1.18M tokens / 415 ms median total cost : $0.0497 same questions for both, clips picked by one fixed rule, not a fact-check

Reply

A Jev-shaped model on Cerebras and Qwen

Built an alternative version of @typesafeai but on @cerebras with Qwen 3.8 27b. Similar quality, similar performance, but vastly different cost. TypeSafe was way cheaper, and did beat Qwen on performance. Closest we can get using LLMs I think. Source: github.com/iammrduncan/ty…

Diogo Almeida
Diogo Almeida
TypeSafe AI
@CompleteSkeptic

After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x

Reply

Jev benchmarked on two public safety corpora

1/ Benchmarked TypeSafe's Jev on two public safety corpora. It doesn't generate text, it returns calibrated probabilities you threshold in code. 96.5% on prompt injection, all 662 messages in deepset/prompt-injections. No tuning. 325ms p50.

Terminal dashboard showing 96.5% accuracy on prompt injection and 89.0% pairwise on vulnerable code, with a context ablation and calibration plot.
Diogo Almeida
Diogo Almeida
TypeSafe AI
@CompleteSkeptic

After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x

Reply