ruvector-typesafe CLINC150 Bank

Install @ruvector/typesafe with npm, download bank.json and questions.json from the Hugging Face repo, then run npx typesafe decide --questions questions.json --bank bank.json --embedder onnx with the engine options used in training.

The bank stores examples and frozen splits, not hyperparameters. Omit the training options and the head refits with library defaults (400 iterations, head: auto), which is a different and materially worse model. Use probeIterations: 4000, probeClassBalanced: true, and head: probe.

On the held-out test split of 4,500 utterances and 150 classes, the bge-small-en-v1.5 encoder reaches 91.0% accuracy with p50 latency of 4 ms and p95 latency of 6 ms. Only bge-small-en-v1.5 was measured for this bank.

CLINC150 includes 1,000 out-of-scope utterances. The bank's abstain mass ranks out-of-scope against in-scope text with an AUROC of 0.9011. Mean abstain is 5.07e-7 for out-of-scope and 2.55e-8 for in-scope, so rank by abstain rather than applying a fixed threshold like 0.35.

importBankJson only admits the examples; the head is fitted lazily on the first decide. At probeIterations: 4000 over 15,000 examples and 150 classes, that fit takes minutes, not milliseconds. Later calls use the steady-state latency. Import once, warm it with one throwaway decision, and keep the engine alive.

It is English only, and both bundled encoders are English sentence encoders. The label set is closed, so new intents need new examples and a refit. Accuracy is reported on this dataset's own test split and is not a claim about your traffic.

The dataset licence is CC-BY-3.0, and this bank redistributes the utterance text under that licence. The @ruvector/typesafe code is MIT. Credit goes to Larson et al., An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction (EMNLP 2019).