JEV Decision Benchmarks

Three: MetaTool tests tool selection and abstention, When2Call tests choosing the right next step (answer, call a tool, ask for missing information, or say it cannot complete), and BFCL V4 uses action and abstention tasks to check calling and refraining.

JEV 1.13 (Choice) reaches 77.79% on similar-tool selection at both 0-shot and 5-shot, 87.04% and 88.54% on abstention, 81.29% on at-most-two tools and 88.33% on exactly-two tools. The public similar-tool and abstention sets each hold 995 examples.

JEV leads on similar-tool selection (77.79% versus 69.05%) and abstention (87.04% versus 50.35%), while ChatGPT is ahead on at most two tools (88.28% versus 81.29%) and roughly ties on exactly two.

It tests the next-step choice among answering, calling a tool, asking for missing information or declining. JEV scores 74.84% accuracy and 56.55 macro F1 across 3,652 examples, but hallucinates a tool call on 197 of the 258 no-tool cases requiring cannot_answer.

JEV uses a Choice adapter that reads four candidates together and selects one, while paper baselines generate tool names followed by answer extraction and official MCQ scoring evaluates candidate continuations separately. The README also marks ordinary accuracy as recoverable for only six official models.

CSV files sit under data/metatool and data/when2call, with Chinese reports at reports/metatool.zh-CN.md and reports/when2call.zh-CN.md and links to the MetaTool and When2Call papers. The README itself is bilingual, English and Simplified Chinese.

The README's BFCL V4 section is cut off mid-table in the digest, no license is listed, and the repository is a results report with reproducible CSVs rather than a runnable harness, so treat the tables as published findings and check the linked reports for protocol details.