Do the faster and cheaper figures generalize to my workload?

No. They describe one recorded pair on one dataset over live networks. An earlier pair timed differently, and quality was close with Gemini marginally ahead on star-rating correlation.

Why does Jev use more input tokens yet cost less overall?

Jev repeats rubrics per question so input tokens run higher, but input is cheap and output is free while Gemini pays for both. Slider re-ranking afterwards costs nothing and needs no model.

How do you handle bad JSON and key leakage on the demo?

Invalid Gemini JSON triggers a counted whole-batch retry, and a leak check asserts no page or replay stream contains the TypeSafe key. Hosted keys need explicit opt-in and BYOK keys are never stored.

Can I explore this without paying for live API calls?

Yes. Recorded runs replay on original timestamps with no keys and labeled replay mode, while live races need TypeSafe and Gemini keys and spend real credits.

How do I run the race on my own machine?

Clone the repository, copy .env.example to .env, add TYPESAFE_API_KEY and LLM_API_KEY for the Gemini lane, then run node server.mjs and open http://127.0.0.1:8777 and click Start race. There is nothing to install: zero npm dependencies, Node 20 or newer.

What is being raced, and on what data?

Four columns over 1,000 Android app reviews from the sealuzh/app_reviews dataset across 17 apps. Sentiment is a five-level score, topic is a six-way choice, bug report is a Noul, and churn risk is a four-level score, with star ratings withheld from both models so they are the independent check on sentiment.

Why does re-ranking after the race need no model calls?

Because it works off the stored typed answers. Jev returned 204 distinct churn values across 1,000 rows against Gemini's 4, and 95 distinct bug probabilities against 19, so its priority score has 673 distinct values with a largest tie of 11 rows while Gemini's has 44 with 299 rows tied. Each slider update measured 36-109 ms in headless Chrome, and the check fails above 150 ms.

CommunityJev Practices & Patterns

Jev Column Race

Live race rig that labels 1,000 real app reviews with Jev typed questions against Gemini JSON output on identical batches, then replays recordings and re-ranks rows locally without extra model calls.

See highlights

Open github.com

Stars: 23
Language: JavaScript
License: MIT
Published by: Community
Author: goodrahstar
Last verified: Oct 2026

goodrahstar/jev-column-race

User repository on GitHub

View on GitHub

Jev vs Gemini 3.8 Flash: labelling 1,000 app reviews, 4.1× faster and 7× cheaper

Forks: 3
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.Homepage

About this project

Four AI columns over 1,000 real app reviews, raced live: TypeSafe's Jev against Gemini 3.8 Flash. Both lanes get the same reviews, the same batches, and the same concurrency. Jev answers 80 typed questions per request. Gemini returns one JSON object per request. Jev labelled all 1,000 reviews in 4.6 seconds for $0.023. Gemini 3.8 Flash took 18.8 seconds and $0.158. That is 4.1× faster and 7.0× cheaper in one recorded run pair, with star-rating agreement of ρ 0.80 vs 0.82. Measurements · Design · Read the questions.

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import awesomejev

Works forJev for Classification

Used inJev for Software & DevTools

Languages: JavaScript

Highlights

Same reviews, batches and concurrency for Jev with 80 typed questions and Gemini with one JSON object.
Recorded runs with server-side keys, visitor BYOK isolation, and free timestamp-faithful replays.
Per-request latency, tail, token and cost accounting with validation and counted retries.
Post-race slider re-ranking from stored answers with finer-grained Jev score distributions.

Watch out

One task, one dataset and one recorded run pair over the live network, so timings vary; agreement rates measure model-to-model match on unlabelled topics and bugs, not accuracy. Costs use list prices. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-does-not-play-dice

Recorded experiments on how Jev reports probability rather than whether it predicts well. On a hidden fair die it picked an option with a mean reported probability of 82.9% while accuracy stayed at 19.0% over 400 trials, and Choice lost a stated 45% versus 55% distinction in an input document.