j341nono/jev-prompt-optimization
User repository on GitHub
automatically optimizing the instructions and decision criteria of TypeSafe Jev Choice from labeled data
- Forks
- 0
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Tools & CLIs
Jev Prompt Optimization
Open-source tool that automatically optimizes the instructions and decision criteria of TypeSafe Jev Choice definitions from labeled data, searching with EvoPrompt or GEPA and scoring candidates with a normalized Brier score.
- Stars
- 1
- Language
- Python
- License
- MIT
- Reading time
- 2 min
- Published by
- Community
- Author
- j341nono
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open sourceRuns locallyRequires API key
Highlights
- Optimizes only the instructions and each criteria description; criterion label names such as billing and technical stay fixed.
- EvoPrompt evolves a population (fitness_sample_size=auto uses min(train, max(12, 4 x label_count))); GEPA uses natural-language feedback.
- Primary metric is 1 - Brier/2 on a 0-to-1 higher-is-better scale; accuracy, NLL, gold-label probability, margins and Jev confidence are kept as diagnostics.
- Web UI at localhost:5173 or CLI: jevopt validate and jevopt optimize with --max-metric-calls, --output, --train/--validation and --demo.
- Requires Python 3.12+, uv and Node.js, plus TYPESAFE_API_KEY and OPTIMIZER_API_KEY; each run calls external APIs, so review the metric-call budget.
Quickstart
bash
git clone https://github.com/j341nono/jev-prompt-optimization.git
cd jev-prompt-optimization
cp .env.example .env
uv sync
uv run jevopt webWatch out
Grounded on the truncated README and GitHub metadata reviewed 2026-10-15; the README's Scope section is cut off after one letter and no benchmark numbers are given for the optimizer's gains on real datasets. The fitness sample formula and Brier metric come straight from the README.
Questions about this entry
Short answers drawn from the linked project or page.
An open-source tool that automatically optimizes the instructions and decision criteria of TypeSafe Jev Choice definitions from labeled data, using an evolutionary approach in which GPT serves as the mutation operator.
Only the instructions and the description of each criteria entry. Criterion label names, such as billing and technical, remain fixed, so the label set you define is preserved throughout the search.
EvoPrompt, a genetic algorithm that keeps promising solutions in a population and evolves it as a whole, treating the several text fields as one group, and GEPA, which uses natural-language feedback about why a prediction was correct or incorrect.
With the normalized Brier score: the Brier score is the sum over labels of (p_k - 1[k = g]) squared, and the primary score is 1 - Brier/2 on a 0-to-1 higher-is-better scale, with accuracy, NLL, margins and Jev confidence as diagnostics.
Python 3.12 or later, uv and Node.js, plus API keys for TypeSafe Jev (TYPESAFE_API_KEY) and the optimizer LLM (OPTIMIZER_API_KEY) in a .env file. Runs call external APIs, so check your dataset and metric-call budget first.
jevopt validate --prompt prompt.json --dataset dataset.json checks inputs, and jevopt optimize --optimizer gepa --max-metric-calls 200 --output optimized.json runs a search; omitting --optimizer selects EvoPrompt, and --train/--validation files can replace --dataset.
The README documents a --demo flag that checks the integration path without API keys, but it explicitly does not simulate Jev quality, so it only verifies plumbing rather than giving meaningful optimization results.
More Jev resources like this
A Rust CLI that streams Kubernetes logs, triages them with local offline rules or typed Jev questions online, and presents an interactive terminal UI plus JSON reports. Collection is read-only and every judgment is advisory.
Jev Tools & CLIs#community#catalog-import#logicrw
Community
sunil-sadasivan/jevernetesGo
sunil-sadasivan/jevernetesGoMaykana's free tool that lets a cheap AI test a web app like a person: write checks in plain sentences, Jev picks the next step from a short list, and below 80% confidence it stops and hands over to a human instead of clicking. A DeepSeek vision pass reports visual defects.
Jev Tools & CLIs#community#catalog-import#logicrw
Community
elshinq/jevaluateJavaScript
elshinq/jevaluateJavaScriptTurns an inbox into a short action queue: Jev answers seven typed questions per email thread in one request, plain Python rules decide what enters the queue, and cached answers mean changing a rule never needs new API calls.
Jev Tools & CLIs#community#catalog-import#logicrw
Community
tusharck/jev-inbox-queuePython
tusharck/jev-inbox-queuePythonA small desktop console and CLI for Jev: paste a state, add noul, choice and score questions by hand, and read calibrated probabilities back without writing code. Standard library only, with a tkinter window whose labels are in Chinese.
Jev Tools & CLIs#community#catalog-import#logicrw
Community
wenchenxi/jev-consolePython
wenchenxi/jev-consolePythonLatitude is an open-source observability platform for AI agents: one line of telemetry captures traces, failing traces group into tracked signals, and fixes are dispatched to coding agents and replayed against real traces.
Jev Tools & CLIs#community#catalog-import#logicrw
Community
latitude-dev/latitude-llmTypeScript
latitude-dev/latitude-llmTypeScriptA local, open-source personal AI assistant for your job: mail, chats and tickets become tasks, agents you already use such as Claude Code, Codex and Gemini do the work, and nothing is sent until you approve. The current release is v0.3.8.3.
Jev Tools & CLIs#community#catalog-import#logicrw
Community
ldbumble/taskuaryPython
ldbumble/taskuaryPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.