codeitlikemiley/system-one-adapter-rust
User repository on GitHub
Rust port of TypeSafe system-one-adapter (LLM-backed system_one evaluations)
- Forks
- 0
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
CommunityJev Practices & Patterns
System One Adapter Rust
Rust port of the Python system-one-adapter: a drop-in stand-in for TypeSafe's `system_one` evaluation API backed by LLM providers instead, so the same questions can be scored against an LLM on cost, speed and intelligence.
- Stars
- 0
- Language
- Rust
- License
- MIT
- Published by
- Community
- Author
- codeitlikemiley
- Last verified
- Oct 2026
About this project
A drop-in replacement for TypeSafe's systemone evaluation API, backed by LLM APIs instead of TypeSafe. Useful for comparing TypeSafe against an LLM on cost, speed, and intelligence. This is a Rust port of the Python system-one-adapter package. Compatible question, answer, retry, and error types are implemented locally — this crate does not depend on typesafe-ai. Provider HTTP clients are optional features (both are enabled by default):
Quoted from the project's README. Read the full README.
- Languages
- Rust
Highlights
- Question, answer, retry and error types are implemented locally, so the crate carries no `typesafe-ai` dependency.
- `llm_attempts` records every provider call in order with messages, schema, response and finish reason, and replays back through `provider.request`.
- Usage adds `input_tokens_total`, `n_retries`, `n_retries_malformed_structure` and `latency` on top of the SDK's existing fields.
- OpenAI's own endpoint uses the Responses API with strict JSON Schema, while custom base URLs default to Chat Completions unless you pass `api`.
Watch out
This is a comparison harness, not a Jev client. The README frames it as a drop-in replacement for the `system_one` evaluation API backed by LLM APIs instead of TypeSafe, useful for comparing the two on cost, speed and intelligence, and the crate implements the compatible types locally rather than depending on the vendor SDK. The blocking client runs its own Tokio runtime when a built-in HTTP provider is used. A response that hits the Anthropic output token limit raises `TypeSafeError` and does not consume malformed-output retries, so the default 4096-token ceiling can surface as an error on larger evaluations. Transient retry count and time budget apply separately to each provider request, while corrective requests share the `n_retry_malformed_structure` allowance and preserve earlier responses and correction messages. Serialization goes through serde; there is no `model_dump`. Copy reviewed against the repository README on 2026-10-04, not from running it.
Questions about this entry
Short answers drawn from the linked project or page.
Comparison is the point. The README describes it as a drop-in replacement for TypeSafe's `system_one` evaluation API, backed by LLM APIs instead of TypeSafe, so you can measure the two against each other on cost, speed and intelligence over the same questions. It is a Rust port of the Python package, and the question, answer, retry and error types are implemented locally, so the crate does not depend on `typesafe-ai` at all.
Pass a provider instance instead of a model string. `provider` is required unless `model` is a provider instance, for example a custom OpenAI-compatible endpoint built with `OpenAIProvider::new("grok-4").base_url(...)`. OpenAI's own endpoint uses the Responses API with strict JSON Schema for structured output, while custom endpoints, including `OPENAI_BASE_URL`, default to Chat Completions. Pass `api` as `responses` or `chat_completions` to force the choice, which is the case the README raises for proxies.
`n_retry_malformed_structure` controls corrective retries, and those requests preserve the earlier responses and the correction messages. Every provider call is recorded in `response.debug["llm_attempts"]` in order, including transient failures and malformed responses, with the messages, the schema and structured flag from `model_request_parameters`, the provider response and the finish reason. You can replay an attempt by deserializing those messages and calling `provider.request` on the same provider. Terminal `TypeSafeError` values expose the same history under `error.debug`.
Partly, and the README is explicit about the additions. The `answers` map and the typed views `nouls`, `choices` and `scores` match the TypeSafe SDK. On top of that, `response.usage` adds `input_tokens_total` and `output_tokens_total` across retries, `n_retries`, `n_retries_malformed_structure` and `latency`. Separately, transient retry count and time budget apply to each provider request independently, so a retry-heavy run reports more total tokens than a single-attempt one.
cargo add system-one-adapter enables the OpenAI and Anthropic provider HTTP clients by default. You can narrow with --features openai only, or --no-default-features --features anthropic. The crate is a Rust port of the Python system-one-adapter and does not depend on typesafe-ai.
The client is configured with an AnswerMode (probabilities for a per-label distribution, or discrete for one value per question) and options such as structured_outputs to use the provider's native structured output, normalize_probabilities to rescale invalid distributions, and n_retry_malformed_structure for corrective retries when schema validation fails.
The default output ceiling is 4,096 tokens; a response that reaches it raises TypeSafeError with instructions to raise max_tokens or ask fewer questions, and it does not consume malformed-output retries. You raise it on the provider, for example AnthropicProvider::new with max_tokens set to 8192.
More Jev resources like this
24GitHub stars
Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
rorshopping/jev-on-a-laptopPython
rorshopping/jev-on-a-laptopPython16GitHub stars
Vendor-neutral harness that tests whether Jev reranking improves a small RAG stack, using frozen hybrid BM25 plus BGE-M3 candidates and isolating retrieval, rerank, and generation stages.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
erendikmenn/jev-rag-benchmarkPython
erendikmenn/jev-rag-benchmarkPython7GitHub stars
Benchmark harness that tests whether models can forecast federal motion-to-dismiss outcomes from the written record, scored per claim-defendant pair with micro-Brier and clustered intervals.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
johnhughes3/LegalForecastBenchPython
johnhughes3/LegalForecastBenchPython6GitHub stars
Frozen 100-question-per-cell check of TypeSafe Jev on Korean versus English reading, paraphrase, and medical exam items, with recorded responses and uncertainty intervals.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
mahlernim/jev-korean-benchmarkPython
mahlernim/jev-korean-benchmarkPython7GitHub stars
Word-level language model experiment that uses Jev as its output layer, with explicit tokenizer, sampler, repetition mask, stop rules, and chunk verification.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
y0usaf/jev-lmTypeScript
y0usaf/jev-lmTypeScript7GitHub stars
Python client and decision toolkit for the Jev System One endpoint with noul, choice, and score primitives, batched asks, composition helpers, concurrent evaluation, and an MCP server.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
docxology/daf-jevPython
docxology/daf-jevPythonFrom the Jev guides
Original write-ups on what this entry is for.
Most Jev accuracy problems are state problems. What belongs in a state, what is a pre-made judgment to leave out, and why naming the slice you ask about matters more than the wording.
Jev Practices & Patterns#state#design#primitives
Read article
A Noul returns one number and no separate confidence. How to read the value, pin its boundary with criteria, split compound questions, and set a per-action threshold that matches the cost of being wrong.
Jev Practices & Patterns#noul#primitives#confidence
Read article
A Score is a weighted position, not a ruler. Why the number can land between levels, why levels are judged separately, and how to combine several atomic scores in code instead of one crowded rubric.
Jev Practices & Patterns#score#primitives#scoring
Read article
From the Jev community
Posts from builders shipping with Jev right now.