Is this evidence Jev is good at reasoning, or a chart chosen to flatter it?

Read the accuracy claim carefully: 70.7% is mid-pack among the 111 models. What the README leads with instead is that Jev was the fastest and second-cheapest at $0.000029 and 0.35 s per question, and that its stated confidence tracks observed accuracy with an expected calibration error of 7.5 pp, which makes it usable as a routing signal even where the raw score does not lead.

Does a 7.5 pp calibration error mean I can use the confidence number as a threshold?

Not on this evidence alone. It is expected calibration error measured over 567 ThaiExam questions for one model version, which bounds neither per-domain nor per-language behavior, and the README recommends no threshold. What would matter for a threshold is the distribution of errors around your cut point, and the README documents neither that nor a class-wise breakdown.

Can I reproduce these numbers, or is it just a static page?

Not from this repository. Both pages are single self-contained HTML files with the data inlined and Chart.js pulled from a CDN, with no build step and no harness. Reproduction means using dtinth/thaiexamjs against the ThaiExam and OpenThaiGPT evaluation datasets yourself; this repo contributes the charts and the credits for the underlying run.

Are the Thai pages a translation, or is the underlying analysis different?

Both pages carry an English and ไทย toggle, so the chart text switches language. The underlying data is the same inlined dataset in both cases; the README does not describe any Thai-specific analysis, prompt handling or scoring difference behind the toggle.

What do the two chart pages show, and do they use the same questions?

The cost-speed-vs-accuracy page scores 111 models on the same 375 questions with thinking models highlighted; the calibration page covers 567 questions. The two pages therefore use different question counts, and the wider run is the calibration one.

Where do the questions and scores come from, and who ran the benchmark?

This repository is only the visualization. The harness is dtinth/thaiexamjs, the questions come from the ThaiExam dataset (SCB 10X) and the OpenThaiGPT evaluation dataset, and the original data and results live at ai-vs-thai-exams.pages.dev.

Will the charts update when a newer Jev version is released?

No. The run is named jev-1.13.0 and nothing here re-runs when the alias moves, and the pages are single static HTML files with the data inlined, so a new model version needs a fresh evaluation and a rebuild.

CommunityJev Practices & Patterns

Thaiexam Jev Charts

Static chart pages comparing Jev against 110 other language models on Thai standardized exams: cost and speed plotted against accuracy, plus a page on whether its stated confidence predicts correctness.

See highlights

Open github.com

Stars: 0
Language: HTML
Published by: Community
Author: vehas
Last verified: Oct 2026

vehas/thaiexam-jev-charts

User repository on GitHub

View on GitHub

Charts: TypeSafe Jev evaluated on Thai standardized exams vs 110 other models

Forks: 0
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

Static chart pages from evaluating TypeSafe's Jev (jev-1.13.0), a System One model, on Thai standardized exams alongside 110 other language models. Both pages have an English / ไทย toggle. Each is a single self-contained HTML file — data inlined, Chart.js from a CDN, no build step.

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import awesomejev html

Works forJev for Classification Jev for Evaluation

Used inJev for Software & DevTools

Highlights

Jev answered 567 questions at 70.7% accuracy, 0.35 s and $0.000029 per question, the fastest and second-cheapest in the comparison.
Expected calibration error of 7.5 pp over those questions, so the confidence number tracks observed accuracy rather than drifting from it.
Two pages only: 111 models on the same 375 questions with thinking models highlighted, and calibration over 567 questions.
Each page is a single self-contained HTML file with inlined data and Chart.js from a CDN, so there is no build step.

Watch out

This repository is the visualization, not the experiment. The benchmark harness is dtinth/thaiexamjs, the questions come from the ThaiExam dataset and the OpenThaiGPT evaluation dataset, and the original data and results live at ai-vs-thai-exams.pages.dev. The headline numbers are for one model on one exam set: 70.7% accuracy is mid-pack in the comparison, so the argument being made is about confidence as a routing signal rather than raw score leadership. The run named jev-1.13.0, and nothing here re-runs when the alias moves. The README does not document the question breakdown, the scoring or grading rules, how per-question cost was computed, how thinking models were detected or given budget parity, or the confidence distribution behind the 7.5 pp figure. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

kyotsu-ai-bench

A static HTML dashboard, four commits deep, scoring Jev against two luna variants on Japan's 2026 Common Test. The repo is results only: CSV tables, one JSON file and two HTML pages, with no scripts to regenerate them.