Using Jev for evals in Datadog

It is a walkthrough. No agreement score, accuracy figure, latency or cost number appears anywhere in it. What you get is a repo to clone, a requirements file, and three notebooks named 1-jev-rubric, 2-online-evals and 3-experiments under DataDog/llm-observability.

Jev is the judge. A single rubric scores every criterion in one Jev request, and the scores are then recorded with LLMObs.submit_evaluation for live spans, or through LLMObs.experiment and BaseEvaluator for offline experiments. The agent trace being scored still needs an OpenAI key.

Not according to the post: there are no preview builds and no private endpoints to point at. It requires ddtrace 4.5.0 or newer, a TypeSafe key for the rubric notebook, and OpenAI plus Datadog credentials for the agent trace.

Datadog's own advice is to measure agreement with human reviewers and repeatability on your own traffic before relying on it. That matters because the rubric is one authored configuration rather than a validated judge, so a plausible-looking score on a span is not the same as a correct one.

By making it an option. A Choice question always returns the highest-probability option, so Jev never abstains on its own, and a cannot-judge outcome has to exist in the criteria. The example rubric adds an unclear option to failure_mode, used when the reply is too short or garbled to judge, and TypeSafe's own moderation cookbook uses the same pattern.

Because the thresholds were calibrated against that exact version and an alias moves when a release ships. Every metric is also tagged judge_model, since the response always reports the versioned model that answered, and logging it lets you compare two judge models on the same spans under the same rubric.

The traced application never imports Jev: a separate worker scores out of band, and Datadog's external evaluations join the verdict to its span by a tag such as turn_id, so production can be scored asynchronously and history backfilled the same way. Passing the turn's own timestamp_ms keeps reruns idempotent. In an experiment the judge function is reused unchanged and cached once per row, so six evaluators over ten rows cost ten requests rather than sixty.