Can WaterSheep replace the hosted Jev API in my stack?

For the typed questions it supports (noul, choice, score, multi), yes: run watersheep --serve and point TYPESAFE_BASE_URL at the local port; any API key value works locally. It is an independent model, not an official Jev endpoint.

How well calibrated are its probabilities?

The README reports 0.026 expected calibration error on the in-distribution test split and 0.043 on held-out datasets, but warns that probabilities are calibrated on training-like data and should be validated on your own.

What does WaterSheep need to run?

pip install transformers torch, then load it with transformers.pipeline(model="samratduttaofficial/WaterSheep", trust_remote_code=True). It is English only and long inputs are truncated, and it needs no API key to run locally.

Which question types can WaterSheep answer?

noul (yes/no), choice (single label), score (a digit scale such as 1 to 5) and multi (multi-label). Every answer includes a probability for each option, and multi returns every option above the threshold.

Can I run WaterSheep in a browser or without Python?

Yes. A JavaScript module runs in the browser with no install, imported as decide from watersheep.js. Other languages can run onnx/model_quantized.onnx with ONNX Runtime, and watersheep.js shows the input format.

How accurate is WaterSheep on data it has not seen?

It reports 77.8% accuracy with 0.026 expected calibration error on its in-distribution test split, and 61.2% accuracy with 0.043 ECE on held-out datasets not seen in training. Per-question-type temperatures are fitted on a validation split.

Can I download WaterSheep for offline use?

Yes. Run hf download samratduttaofficial/WaterSheep --local-dir WaterSheep, or git clone the repository with Git LFS, then load the folder with pipeline(model="WaterSheep", trust_remote_code=True) without network access.

CommunityJev Models & Reimplementations

WaterSheep

Apache-2.0 ModernBERT fine-tune that answers noul, choice, score and multi-label questions with a probability for every option. It serves Jev's POST /v1/systemone locally so TypeSafe's Python SDK runs unchanged, reporting 0.026 expected calibration error on its test split.

See highlights

Open github.com

Stars: 12
Language: Python
License: APACHE-2.0
Published by: Community
Last verified: Oct 2026

SamratDuttaOfficial/WaterSheep

User repository on GitHub

View on GitHub

A decision model that returns calibrated answers to yes/no, choice, score and multi-label questions.

Forks: 3
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.Homepage

About this project

WaterSheep answers yes/no, single-choice, rating and multi-label questions about any text, with a probability for every option. Every answer includes a probability for each option. WaterSheep is an open-source alternative to Jev. Run it as a local server: It answers Jev's POST /v1/systemone requests on your machine, and TypeSafe's Python SDK works against it without code changes:

Quoted from the project's README. Read the full README.

Tagscommunity calibration modernbert onnx

Works forJev for Classification Jev for Scoring

Used inJev for Customer Support

Languages: Python JavaScript
Platforms: API Browser
Capabilities: Open sourceSelf-hostedRuns locally

Highlights

Answers yes/no, single-choice, rating-scale and multi-label questions with a probability for each option
Serves POST /v1/systemone on port 8766; set TYPESAFE_BASE_URL and TypeSafe's Python SDK runs unchanged
In-distribution test split: 77.8% accuracy with 0.026 expected calibration error
Held-out datasets: 61.2% accuracy and 0.043 ECE, with per-question-type temperature calibration
Ships Python, browser JavaScript and ONNX Runtime paths, with weights downloadable from Hugging Face

Watch out

Apache-2.0. English only, long inputs are truncated, and the author states it is not for high-stakes medical, legal, financial or hiring decisions. Independent of and not affiliated with TypeSafe AI.

Questions about this entry

Short answers drawn from the linked project or page.

Related terms

Glossary definitions related to this entry.

More Jev resources like this

Gutsy

An Apache-2.0 0.8B decision model that returns calibrated probabilities for yes/no, choice and score questions on CPU. Distributed as GGUF for llama.cpp and served behind a Jev-style /v1/systemone API, so existing Jev clients mostly need only a new base URL.

Jev Models & Reimplementations#community#gguf#local-inference

Community

kouhxp/gutsyHTML

AutoJev

130

An open recipe for a 27B multimodal decision model trained with full-weight SFT on 73,000 examples over one H200, serving choice, noul and score on /v1/systemone with per-checkpoint provenance and released calibration plots.

Jev Models & Reimplementations#community#calibration#qwen

Community

denis-pplx/autojevPython

ruling

A local Jev-compatible server that reads Choice, Score and Noul answers from the logits of any MLX checkpoint or OpenAI-compatible endpoint with no training, serving POST /v1/systemone as a drop-in for the official TypeSafe SDK.

Jev Models & Reimplementations#community#mlx#calibration

Community

bradAGI/rulingPython

Luce

Open recipe for calibrated decision models: an LLM teacher writes training data from a one-sentence task description, a LoRA plus decision head on Qwen3-4B-Base answers Choice, Score and Noul questions with calibrated probabilities, evaluated next to Jev on identical items.

Jev Models & Reimplementations#community#calibration#training

Community

scienthoon/lucePython

Laya Typed-Decisions

The fine-tuned Laya checkpoint for four typed-decision workflows: 0.766 accuracy on 2,000 held-out decisions, above Jev 1.13.0's published 0.727 and the teacher self-agreement ceiling it trained past.

Jev Models & Reimplementations#community#open-models#local

Community

Rizzo Flow

877

A local server that reads next-token probabilities from a fine-tuned Spark model and returns Choice, Score, and Noul answers with zero generated tokens.

Jev Models & Reimplementations#community#python#open-models

Community

Rizzo-AI-Academy/rizzo-flowPython

From the Jev guides

Original write-ups on what this entry is for.

Multimodal Decision Models: What Clef and d1 Change

Oct 12, 2026

Jev takes text only; Clef-omni adds image, audio, and video input and Liquid d1 adds images, each keeping the same typed-answer shape. Where that matters, where it does not, and the costs the vendors underplay.

Jev Models & Reimplementations#multimodal#open-models#decision-model

Read article

The Decision-Model Price War

Oct 11, 2026

One model in September, a dozen by October. Clef, Liquid d1, Perplexity, and OpenAI all ship the same typed-answer shape now. How to read the rival benchmarks and pick on your own task.

Jev Models & Reimplementations#decision-model#benchmarks#pricing

Read article

What Is Actually on Hugging Face

Oct 8, 2026

Hundreds of repositories carry the system-one tag, but the ecosystem is a dozen ideas in different sizes. A tour of the families, the licenses, and what a download does not include.

Jev Models & Reimplementations#open-models#hugging-face#local

Read article

Back to all resources

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai