Question 1

Are the benchmark numbers in that table real, or is it a ranking?

Accepted Answer

Real runs, meaningless sample. The README states every baseline cell came from running scripts/run_baseline.py and routes all three models through OpenRouter with provider data collection set to deny, so no row went to a provider that keeps prompts. It then immediately qualifies them: the row counts are 5, 5 and 7 seed rows, not labeled production rows, the export found zero human-labeled rows per decision, and at n=5 a single row is worth 20 points of accuracy. Read the columns as the runner working and the shape of the task, not as a leaderboard.

Question 2

Why is calibration the headline metric rather than accuracy?

Accepted Answer

Because of what the software does with the answer. PadFlow posts automatically above a threshold and queues the rest for a human, so what matters is whether the model is right on the rows it claims to be confident about. The README's framing: a model that is 85% accurate but knows which 60% it is sure of is more useful than one that is 90% accurate and always says 0.95. The default auto-post threshold is 0.90.

Question 3

How much labeled data is actually public today?

Accepted Answer

Seed rows only. The design is a split of 25 rows per decision published with the rest kept private, and the baseline is meant to be measured on the full set, but the files hold seed rows only right now. data/README.md is where the counts and the reason are kept, and the README's own summary is that the set will be enough to rank models and is not enough to certify one.

Question 4

Can I score my own model against this workload?

Accepted Answer

Yes, that is the point. run_baseline.py talks to any OpenAI-compatible chat endpoint that does JSON schema output, and --base-url points it elsewhere; --model and --decision narrow the run and --data-dir private uses the unpublished rows. Results land in out/<model>.json and the summary prints as a markdown row you can paste into the table. The documented invocations expect an OpenRouter API key, and each decision is a typed input, an enum label and a confidence, which is the shape any decision model has to answer in.

Question 5

Which three decisions does the benchmark cover?

Accepted Answer

route_document, which project and document kind an email or file is; code_transaction, which budget line and draw a synced QuickBooks transaction belongs to; and classify_import_value, what a value read from an imported schedule or budget page is. Each schema has the same shape: an input object, a decision enum, and a confidence in [0, 1]. PadFlow posts automatically above a threshold and queues the rest for a human.

Question 6

How is the public data anonymized, and what is kept?

Accepted Answer

Company names, people, project names, emails, phone numbers, street addresses and account numbers are replaced with stable placeholders such as Company_A (grading), Person_3, Project_1 and a placeholder street. Amounts, dates, line-item descriptions and document structure are kept, because those are what the decision depends on. The baseline runs route all three models through OpenRouter with provider data collection set to deny, so no row goes to a provider that keeps prompts.

Question 7

What does the cost side look like, and what flags does the runner take?

Accepted Answer

Install requirements, export OPENROUTER_API_KEY, then run scripts/run_baseline.py with --model, an optional --decision to narrow to one, --base-url to point at another OpenAI-compatible endpoint, and --data-dir private for unpublished rows. Results land in out/<model>.json and print a markdown row. For context, PadFlow's September 2026 page-reading trial ran $3 to $33 per 1,000 pages, and the README says per-decision price is the constraint because decisions happen far more often.

Padflow Jev Evals

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev guides

From the Jev community