Are the accuracy numbers real measurements?

They are observed counts on fixed synthetic tasks with published payloads and per-question token files, not an official model score. The README stresses comparisons across rows are different tasks, not one leaderboard.

Does passing the simple task prove the harder one works?

No. The central finding is that correct prerequisite answers do not always produce correct decisions, and passing a simple task does not establish reliability on a harder version of it.

Why publish a driving pilot that finished 0/12 tasks?

It is evidence about framing: direct steering chose straight in all 522 steering calls and finished 0/12 full tasks, while paused versus real-time and assisted conditions are compared with replays.

Can I reproduce these runs or only read the reports?

Both. The site replays 96 recorded Snake games and 24 city episodes, and separate guides show how to run the Snake controller and city demo live locally with new API calls.

Does the position of the correct option matter?

In the arithmetic permutation study accuracy was 95/108 when the answer was first, falling to 67/108, 64/108 and 62/108 as it moved later, so choice order is an evaluation variable rather than harmless formatting.

What made the Snake interface better?

The best tested interface describes the board in plain language with explicit priorities, reaching 14/16 held-out targets against the original prompt's 1/16 and 90.0% shortest moves; history, ASCII grids and a second self-review call showed no benefit in screening.

Does breaking a decision into two steps help?

All three methods tied at 288/288, but the check-prerequisite-then-choose path cost far more: mean input tokens of 808.63 against 365.81 for the direct question, and 74.50 output tokens against 32.50.

CommunityJev Practices & Patterns

Jev Behavior Study

Independent browsable field guide to Jev 1.13.0 covering 11,621 text-study requests, Snake controllers, and a 3D-city driving lab with replays, token counts, and reports.

See highlights

Open github.com

Stars: 3
Language: Python
License: MIT
Published by: Community
Author: RINNECODER
Last verified: Oct 2026

RINNECODER/jev-behavior-study

User repository on GitHub

View on GitHub

Independent Jev 1.13.0 behavior study: report, controlled prompt experiments, raw results, and offline verification.

Forks: 0
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

A browsable field guide to Jev 1.13.0: where explicit questions work, where small changes alter answers, and where harder tasks expose failures. 11,621 text-study requests · 3 Snake studies · 3D City lab · 7 detailed reports. Live API observations from September 16–17, 2026 (UTC). Independent, AI-assisted research. Explore findings · How to interact with Jev · Tokens per question · Inspect the evidence · Reproduce.

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import awesomejev

Works forJev for Classification

Used inJev for Software & DevTools

Languages: Python

Highlights

Decision panel ties at 288/288 across direct and two-step methods, with the two-step path costing roughly double the tokens.
Arithmetic accuracy shifts with answer position, scoring higher when the correct choice appears first.
Unassisted Snake screening favors plain-language board descriptions, reaching 14/16 held-out targets.
City-lab viewer replays 24 pilot episodes with camera controls, scrubbing, and per-question tokens.

Watch out

Synthetic task-specific observations, not an official benchmark or overall model score, and repeated calls are not independent new problems. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-on-a-laptop

Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.