Question 1

Are those benchmark numbers comparable to TypeSafe's own?

Accepted Answer

They come from this repository's own evaluation runs, under a stated protocol: JevBench v1.2 plus 13 human-annotated public subsets totalling 3,880 items, matched to the Bespoke Nimble setup. The table places Jev 1.13.0 at 76.0% and 75.3% as a reference, with Bespoke Nimble-9B at 74.8% and 63.5%. Useful as a self-consistent comparison, but nobody has audited the harness here.

Question 2

Is this a drop-in replacement for the TypeSafe API?

Accepted Answer

No, and nothing here claims it is. It is an open-weight model you run yourself behind a candidate-logit scorer: you pass state plus a schema of choice, boolean or rubric score and get calibrated probabilities per option, with no generation step and no parsing step. Code is Apache-2.0, but the weights follow the Qwen3.5 base terms, which is the part to read before deploying.

Question 3

Why is only one of the two listed models actually available?

Accepted Answer

The 4B weights are published on Hugging Face as a policy mix. The 0.8B row is explicitly marked as not released, and the ModelScope publication for the 4B is marked as pending. The release directory holds the dual-publish pipeline, so the second channel is wired up but not filled in.

Question 4

Can I reproduce the training, or only the inference?

Accepted Answer

Inference, yes: install the requirements and run the demo script against the published checkpoint. Training, only partially. The recipes directory has recipe cards and reproducible train and eval scripts, and data_pipeline has the view augmentation and teacher soft-label synthesis code, but the teacher distillation recipe for the training data is internal documentation, so the dataset itself is not reproducible from this repository.

Question 5

How do I run inference with it?

Accepted Answer

Clone the repository, install inference/requirements.txt, then run python inference/demo.py --model wayfind/metask-jev-4b-policy-mix. Input is a state plus a schema of choice, boolean, or rubric score, and the output is a calibrated probability per option from a single forward pass, about 24 ms per decision on an RTX 4090.

Question 6

What is actually in the repository?

Accepted Answer

Five directories: models holds the registry, model cards, and evaluation results; recipes holds training recipe cards and reproducible train and eval scripts; data_pipeline holds view augmentation and teacher soft-label synthesis; inference is the candidate-logit scorer with MPS and CUDA adaptation; and release is the Hugging Face plus ModelScope publish pipeline.

Question 7

What license covers the code and the weights?

Accepted Answer

The code is Apache-2.0, while the model weights follow the terms of their Qwen3.5 base. That split matters if you plan to deploy the 4B policy-mix checkpoint commercially, so read the weight terms separately from the repository license.

Metask Jev

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev guides

From the Jev community