Langfuse docs: Jev as a decision-model evaluator

An evaluator that uses a decision model such as OpenAI's gpt-6-luna or TypeSafe Jev to answer typed questions about an observation instead of generating free-form text. Each question becomes a Langfuse score, and probabilities and confidence are stored in the score metadata where the provider returns them.

Use a decision model when the verdict is narrow and typed: a label, a level on a rubric, or a yes/no probability. Use LLM-as-a-judge when you need written reasoning, or when the possible answers are too open-ended to define upfront.

Choice picks one option from a fixed set you define, 2 to 255, returning probabilities and confidence where available. Score rates ordered levels, 2 to 10, returning a probability-weighted value. Yes/no returns the probability that a statement is true, which TypeSafe calls a Noul.

OpenAI receives one general state named input, and its questions use plain text that cannot reference parts of the state. Jev receives a JSON state with named fields, and questions can reference those fields explicitly using backticks, for example comparing output with expectedOutput.

It cites TypeSafe's reports that Jev is 40 to 400x cheaper than frontier models on classification tasks and 20 to 200x faster than comparable LLMs, alongside OpenAI's listed price of $0.10 per million input tokens for gpt-6-luna with no output-token charge.

Give every Choice option a short description and an escape hatch such as other or unclear; describe every Score level in plain language and order levels low to high; for Yes/no you can optionally define what counts as true and false, then pick a threshold that fits your risk tolerance.

The docs say to keep one decision per question. If you find yourself asking whether the answer is correct and polite, split it into two questions; both are answered in the same call anyway, and atomic questions align with good evaluator practice.