Can I run the showdown without API keys?

Yes. Any provider without a key runs in clearly labelled MOCK mode with deterministic fake probabilities; pytest also forces MOCK mode so nothing leaves your machine, and mock numbers mean nothing about the models.

Do the models make the decision, or does code?

Plain code makes the decision. Both APIs return probability distributions for typed questions, and your routing rules and thresholds - such as TICKETS_CHOICE_THRESHOLD or ALERTS_PAGE_SEVERITY - decide the outcome.

What are the two apps and their sample sizes?

A support ticket router over 20 hand-labelled tickets for a fictional SaaS, asking team (choice), security_sensitive (yes/no) and customer_impact (score 0-3); and a cloud alert triage over 15 hand-labelled alerts, asking severity (score 0-3), recommended_action (choice) and likely_caused_by_deploy (yes/no). Code routes to a team queue, the security queue or human review, and to ignore, watchlist, page on-call or rollback.

Which providers and endpoints are compared?

OpenAI Decisions with model gpt-6-luna on POST /v1/decisions, and TypeSafe Jev with model jev-latest on https://api.typesafe.ai/v1/systemone. Identical typed questions go to both in parallel, and both are normalized through one DecisionProvider interface so the answers are comparable.

What do the threshold defaults look like?

Everything is optional in .env. Tickets default to a 0.75 team-confidence floor below which a ticket goes to human review and a 0.5 security threshold above which it goes to the security queue; alerts default to paging at expected severity 1.6, watchlisting at 0.7, and rollback at 0.7 when a page-level alert follows a recent deploy. All of them can also be changed per run in the UI.

What does eval.py actually report?

Accuracy against the human labels, the agreement rate between providers, p50 and p95 latency, total cost, and a calibration table bucketed by confidence at under 0.5, 0.5 to 0.7, 0.7 to 0.9 and 0.9 to 1.0. The tickets eval also sweeps thresholds from 0.60 to 0.95 to show how many tickets go to human review versus get misrouted.

How do I set it up and check the keys?

Python 3.11 or newer, a virtualenv, then pip install -r requirements.txt for FastAPI, Uvicorn, openai>=3.26.0, typesafe-sdk and pytest. Copy .env.example to .env and fill in either key; leaving one empty keeps that provider in MOCK mode. `python check_keys.py` then makes one tiny live call per provider and exits non-zero if a live check fails, so it can run in CI.

CommunityJev Demos & Experiments

Decision Models Showdown

Two runnable demo apps that send identical typed questions to OpenAI Decisions (gpt-6-luna) and TypeSafe Jev (jev-latest) in parallel: a support ticket router over 20 labelled tickets and a cloud alert triage over 15 labelled alerts, with thresholds in plain Python making the final calls.

See highlights

Open github.com

Stars: 0
Language: Python
Published by: Community
Last verified: Oct 2026

ivipinmishra/decision-models-showdown

User repository on GitHub

View on GitHub

OpenAI Decisions vs TypeSafe Jev, side by side: two demo apps where decision models answer typed questions and plain code makes the call.

Forks: 0
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

Compare two "decision model" APIs side by side on the same questions: OpenAI Decisions: model gpt-6-luna, POST /v1/decisions. TypeSafe Jev (System One model): model jev-latest, `POST. Both answer typed questions (yes/no, pick one, rate on a scale) with probabilities instead of prose. This project sends identical questions to both in parallel. Plain code then makes the final decision, using thresholds you control. It contains two projects in one web app: What this shows / what it doesn't.

Quoted from the project's README. Read the full README.

Tagscommunity comparison fastapi tickets alerts

Works forJev for Routing Jev for Classification Jev for Scoring Jev for Evaluation

Used inJev for Software & DevTools Jev for Customer Support

Languages: Python
Platforms: Browser API
Capabilities: Self-hosted

Highlights

Both providers answer yes/no, choice, and score questions with probabilities instead of prose, normalized through one DecisionProvider interface.
Tests always force MOCK mode; without API keys either provider runs with deterministic fake probabilities, clearly labelled MOCK in the UI.
eval.py reports accuracy against human labels, provider agreement, p50/p95 latency, total cost, calibration buckets, and a 0.60-0.95 threshold sweep.
Every call is logged to decisions_log.sqlite with input hash, provider, answers, latency, and cost.
check_keys.py makes one tiny live call per provider and exits non-zero if a live check fails, so it can run in CI.

Watch out

No license is declared in the repository; the README calls the project demo code for discussion, with no retries beyond SDK defaults, no auth on web endpoints, and no PII handling.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

Gutsy

An Apache-2.0 0.8B decision model that returns calibrated probabilities for yes/no, choice and score questions on CPU. Distributed as GGUF for llama.cpp and served behind a Jev-style /v1/systemone API, so existing Jev clients mostly need only a new base URL.

Jev Models & Reimplementations#community#gguf#local-inference

Community

kouhxp/gutsyHTML

Jev Playground

An independent browser playground that runs TypeSafe Jev on one question or several, with ready examples for routing, refunds, urgency, and the next agent action.

Jev Demos & ExperimentsPlayground#community#playground#tutorial

Community

AgentScope

33k

Alibaba's multi-agent platform, which implements native TypeSafe Jev classification through JevClassifierModel for binary, choice and score routing across agent pipelines. Python 3.11+, Apache-2.0, with a FastAPI agent service and web UI.

Jev Integrations & Adapters#community#multi-agent#framework

Community

agentscope-ai/agentscopePython

AutoJev

130

An open recipe for a 27B multimodal decision model trained with full-weight SFT on 73,000 examples over one H200, serving choice, noul and score on /v1/systemone with per-checkpoint provenance and released calibration plots.

Jev Models & Reimplementations#community#calibration#qwen

Community

denis-pplx/autojevPython

FastJev

Self-hosted Python SDK and System One-compatible API for runtime-defined Choice, Boolean and Score decisions on pinned open models, across Torch, vLLM, MLX, llama.cpp and WebGPU. Each result records probabilities, token usage, timing, model revision and prompt version, with committed row-level benchmarks.

Jev Repos & SDKs#community#sdk#self-hosted

Community

chengyongru/fastjevPython

WaterSheep

Apache-2.0 ModernBERT fine-tune that answers noul, choice, score and multi-label questions with a probability for every option. It serves Jev's POST /v1/systemone locally so TypeSafe's Python SDK runs unchanged, reporting 0.026 expected calibration error on its test split.

Jev Models & Reimplementations#community#calibration#modernbert

Community

SamratDuttaOfficial/WaterSheepPython

From the Jev guides

Original write-ups on what this entry is for.

Per-Tick Decisions in Games

Sep 28, 2026

A game tick is a state plus a legal action list. How Clash Royale, Minecraft, and the earlier Mario, Pokémon, StarCraft, and drone loops ask Jev one step at a time.

Jev Demos & Experiments#gaming#agents#control

Read article

Back to all resources

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai