Skip to content
JevDirectory.org
CommunityPractices & Patterns2 starsVerified 2026-09-22

jev-research-eval

A reproducible harness that scores Jev Ultrafast research-browser runs over 11 baseline cases plus human and quant stress suites with CoS-locked QC grades, regenerating the field note and trace notebooks offline.

Open github.comjgridifier/jev-research-eval
Category
Practices & Patterns
Published by
Community
Author
jgridifier
Added
2026-09-22
Tagscommunitypythonevaluationbenchmarksbrowseragents

Highlights

  • Drives an upstream browser-use/jev-ultrafast checkout pinned at commit 452c1ad; it is not a fork of Jev.
  • Cases cover 11 baselines (R1-R11), 10 human stress workflows (S1-S10), and nine quant/ops stress cases (QS1-QS8 plus QS7b).
  • The v4 field note and both notebooks regenerate offline from fixtures/qc_rescored.json with no browser or API keys.
  • Live suite re-runs need uv, Chrome with CDP, and TYPESAFE_API_KEY, TEXT_MODEL_API_KEY, and BU_CDP_URL.
  • Not affiliated with Coinbase, Meta/Astryx, Goldman Sachs, or browser-use beyond use of public open-source software.

Quickstart

bash
python scripts/generate_report_v4.py \
  --input fixtures/qc_rescored.json \
  --output /tmp/jev_note_regen.html

Watch out

The repo layout states MIT for the harness only, though GitHub reports NOASSERTION; it is not a Jev fork, and live re-runs require uv, Chrome with CDP, and API keys.

More like this

jev-agent-failure-benchmark
2GitHub stars
A benchmark that runs Jev on all 6,257 text traces of Who&When Pro to attribute agent failures, scoring it with the official pinned scorer against the paper's GPT-5.4, Claude Sonnet 4.6, GLM-5, and Qwen3.5-122B baselines.
Practices & Patterns#community#python#benchmarks
Communityjev-agent-failure-benchmark
jeff
208GitHub stars
A self-hosted implementation of TypeSafe's Jev System One API powered by the 400M-parameter GLiFormer encoder: it serves choice, score, and noul and drops into the official typesafe-sdk via TYPESAFE_BASE_URL, but trails Jev on reasoning-heavy tasks.
Practices & Patterns#community#python#open-models
Communityjeff
jev-eval-agent
104GitHub stars
A personal-assistant agent with 100 mocked tools that measures how many steps a task takes when the LLM picks the tool versus when Jev picks it before every model step.
Practices & Patterns#community#typescript#agents
Communityjev-eval-agent
Back to all resources

From the community

Posts from builders shipping with Jev right now.

Follow @typesafeai

Browser Use Ultrafast, powered by Jev

A really smart switch statement

Nathan Flurry 🔩
Nathan Flurry 🔩
Rivet
@NathanFlurry
·

hype-free explanation of jev: jev does not replace gpt / claude jev is just a *really* smart switch statement like if 2016 ml classifiers got 2026 levels of intelligence it's a new* type of tool that will make a lot of workloads insanely fast, cheap, and accurate * = and by Show more

Diogo Almeida
Diogo Almeida
TypeSafe AI
@CompleteSkeptic

After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x

Watch on X
Reply
Read 230 replies

When a designer gets Jev

Full Jev video tutorial

The case against Jev-scored compaction

Theo - t3.gg
Theo - t3.gg
@theo
·

This is a terrible compaction strategy that fundamentally doesn't understand how compaction and context management work. Seems like a lot of people are confused so let's break this down. 1. Compaction isn't a filter The role of compaction is to clean up history to keep the Show more

tamara
tamara
@tamarajtran

found the perfect use case for @typesafeai Jev: instant compaction in 2026, why is compaction still a summarization prompt? Jev can make it instant by scoring every tool call and dropping what’s irrelevant

Watch on X
Reply
Read 182 replies

Classifying rows in DuckDB