What does the repo run?

Four workflows — invoice processing (150 cases), customer service (204), agent traces (111), and security incidents (240) — against the default typesafe:jev-1.13.0 and other providers including OpenAI, Anthropic, Fireworks, Groq, and Cerebras.

Where do the labels come from?

Inputs and reference labels download from the typesafe WorkflowEvals collection on Hugging Face. The caveat says the workflow code is assumed correct: it compares models and does not re-check the questions.

What do I need installed to run the evals?

Python 3.13 or newer and uv. Then run uv sync --locked, export TYPESAFE_API_KEY, and run a workflow such as uv run python run.py invoice_processing --limit 5.

How do I evaluate a different model in the harness?

Pass --model provider:model with the provider's exact model ID, such as --model openai:gpt-6-astra, and set that provider's API key. Supported providers are openai, anthropic, fireworks, groq, cerebras, and typesafe.

Which metrics does the WorkflowEvals scorer report?

exact_actions, which matches action sets including their arguments, and primary_action, which matches just the primary actions. Not every workflow defines primary actions.

How do I resume or rerun failed cases?

Repeat the same command to resume saved cases, and add --retry-errors to rerun failed cases. Add --dry-run to check inputs without making any model calls.

How can I compare a workflow run against my own model run?

Add --reference pointing at another run folder. Scores then measure agreement with the reference model, and a model is excluded from scoring against its own answers.

OfficialJev Benchmarks & Evaluations

WorkflowEvals

The code behind evals.typesafe.ai. It runs four published workflows against Jev and other providers and scores them on reference labels.

See highlights

Open github.com

Stars: 16
Language: Python
License: APACHE-2.0
Published by: TypeSafe AI
Last verified: Oct 2026

typesafe-ai/WorkflowEvals

Organization repository on GitHub

View on GitHub

evals.typesafe.ai workflow code published

Forks: 1
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

This code allows you to reproduce the results from our evals site. The eval assumes the workflow and its code is correct and evaluates models against the largest models from the most well-known labs. Install Python 3.13+ and uv, then:

Quoted from the project's README. Read the full README.

Tagsofficial evaluation python

Works forJev for Evaluation

Used inJev for Software & DevTools

Languages: Python

Highlights

Four workflows: invoice processing (150 cases), customer service (204), agent traces (111), and security incidents (240).
The default model is typesafe:jev-1.13.0. Other providers listed are OpenAI, Anthropic, Fireworks, Groq, and Cerebras.
Inputs and reference labels download from the typesafe WorkflowEvals collection on Hugging Face.
Apache-2.0. Requires Python 3.13 or newer and uv. 13 GitHub stars on 3 October 2026.

Watch out

The README says the workflow code is assumed correct. It compares models. It does not re-check the questions.

Questions about this entry

Short answers drawn from the linked project or page.

Related terms

Glossary definitions related to this entry.

Precision and recall

More Jev resources like this

hev reranker

A BEIR reranking recipe that asks up to 30 parallel Noul relevance questions in one Jev state, with RESULTS.md against cross-encoders and hosted rerankers.

Jev Benchmarks & Evaluations#community#evaluation#python

Community

hev/rerankerPython

typesafe-jev-calibrate-for-code-review

A calibration write-up and scripts for Jev comment and code rules on a Unity project, with ordering.py to compare rule versions on real blocks.

Jev Benchmarks & Evaluations#community#evaluation#code-review

Community

Selmar/typesafe-jev-calibrate-for-code-reviewPython

Jev on r/AmItheAsshole

A 770-post benchmark that scores Jev, Sonnet 5, GPT-5 nano, and two local models on Reddit verdicts with weighted Brier scores.

Jev Benchmarks & Evaluations#community#python#evaluation

Community

dchristopoulos/jev-aitaPython

Jev calibration study

A sentiment study of whether Jev's stated confidence matches accuracy, comparing raw scores with Platt scaling and isotonic regression.

Jev Benchmarks & Evaluations#community#python#evaluation

Community

AnthusAI/Jev-CalibrationPython

S1MB

System One Mosaic Benchmark: a Python evaluator and Next.js viewer comparing Jev and open decision models across 137 English Choice, Noul and Score benchmarks from 106 dataset subsets, with recorded evaluation conditions and a public Hugging Face leaderboard.

Jev Benchmarks & Evaluations#community#benchmark#leaderboard

Community