Should I stay on jev-latest or pin a version?

Evaluations default to jev-latest, and the README's advice is to pin a versioned id once you have tuned thresholds against it — passing model: "jev-1.13.0" to evaluate. For the model list, bundle exec rake models refreshes models.json from TypeSafe's listing, which returns aliases such as jev-latest; versioned ids like jev-1.13.0 are accepted even when the listing does not mention them.

What does one call cost, and where can I read that?

The evaluation object exposes cost.total alongside model, so per-call cost is readable in code rather than estimated. The README's worked example returns model "jev-1.13.0" and cost.total 0.000016884 for a single support message judged with three questions: a noul, a choice over three teams, and a three-level frustration score.

Can I ask about one field of a structured payload, not the whole thing?

Yes. The state can be a String, a Hash or an Array, and a question addresses a part of it with a backticked path such as `ticket.messages[0].text`. Every question runs in parallel against the same state, so grouping independent questions into one call costs roughly what a single question does.

Why does rerank fall over on my whole document index?

Because the query and all documents share Jev's 32k-token state budget. Reranking gives every document a relevance question in a single request and scores it by the probability that it helps answer the query, which is fast per document and unbounded in total. Rerank a shortlist from a faster search rather than a corpus.

How do I install and configure the provider?

Add gem 'ruby_llm-providers-typesafe' and set config.typesafe_api_key in a RubyLLM.configure block, usually from the TYPESAFE_API_KEY environment variable. Outside Bundler you must load it yourself with require 'ruby_llm/providers/typesafe'.

What does an evaluation return for each question?

The evaluation object returns one typed answer per question: probability and yes? for a noul, option, probabilities and confidence for a choice, and score plus level for a score, along with the answering model and cost.total.

What is rerank for, and what does top_n do?

RubyLLM.rerank gives every document a relevance question in one request and scores it by the probability that it helps answer the query. top_n keeps the best matches; rerank a shortlist from a faster search rather than a whole corpus, because the query and documents share the 32k-token state budget.

CommunityJev Practices & Patterns

Ruby LLM Providers TypeSafe

RubyLLM provider that turns typed questions — noul, choice and score — into calibrated answers with probabilities, confidence and per-call cost, and adds reranking through RubyLLM.rerank.

See highlights

Open github.com

Stars: 4
Language: Ruby
License: MIT
Published by: Community
Author: javiergradiche
Last verified: Oct 2026

javiergradiche/ruby_llm-providers-typesafe

User repository on GitHub

View on GitHub

TypeSafe System One models (Jev) for RubyLLM: typed judgments, evaluations and reranking.

Forks: 0
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.Homepage

About this project

TypeSafe for RubyLLM. TypeSafe's System One models, starting with Jev, answer typed questions about your data with calibrated probabilities instead of generated text. Outside Bundler, load it with require 'rubyllm/providers/typesafe'. Ask questions about a state, and get one typed answer per question: noul asks a yes/no question. Its answer is the probability of yes. Pass yes: and no: to describe what each answer means.

Quoted from the project's README. Read the full README.

Tagscommunity awesomejev catalog-import

Works forJev for Classification Jev for Ranking Jev for Evaluation

Used inJev for Software & DevTools

Languages: Ruby

Highlights

One evaluate call asks every question against the same state in parallel and returns probability, option, score, level, model and cost.
State can be a String, Hash or Array; questions address parts with backticked paths such as `ticket.messages[0].text`.
yes? compares the probability to 0.5 by default, takes yes?(threshold: 0.8), and confidence is there to send uncertain ones to a person.
Rerank puts every document's relevance question into one request, but the query and all documents share a 32k-token state budget.

Watch out

Evaluations default to jev-latest, an alias, so pin a versioned id such as jev-1.13.0 once your thresholds are tuned or a model change will move them underneath you. Reranking is not for whole corpora: the query and every document share Jev's 32k-token state budget, so shortlist from a faster search first. A score can land between levels, so treating it as a rubric boundary is your problem to handle. Outside Bundler you must require the provider yourself. The first local test run calls the real API to record VCR cassettes, and CI only replays cassettes already committed. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-rag-benchmark

Vendor-neutral harness that tests whether Jev reranking improves a small RAG stack, using frozen hybrid BM25 plus BGE-M3 candidates and isolating retrieval, rerank, and generation stages.

Jev Practices & Patterns#community#jevbest#catalog-import

Community

erendikmenn/jev-rag-benchmarkPython

jev-carryforward

An MCP server with two tools for agent memory: record writes facts into a per-project ledger as they happen, and recall scores each one against the task you are starting and hands back only what matters.