Skip to content

Jev directory

Jev Decoder

A set of experiments that uses TypeSafe's Jev as a decoder through its Choice primitive, sampling ASCII characters, BPE tokens and grammar-constrained SQL from Jev's probability distributions, and measuring next-character loss against uniform sampling on tiny-shakespeare.
Stars
0
Language
Python
Reading time
2 min
Published by
Community
Author
panalexeu
Last verified
Oct 2026

Highlights

  • ascii_char_gen and token_gen sample characters or BPE tokens from Jev Choice; Choice takes at most 255 options, so the tokenizer vocabulary is capped.
  • char_eval measures next-character loss over a 256-character window: 7.71 nats versus 4.17 for uniform sampling, with 45 catastrophic misses above 20 nats.
  • sql_gen builds SQL from a Lark grammar plus NUMBER and STRING tokens taken from the reference query; 489 of 1,034 Spider dev queries matched the reference.
  • Setup uses uv scripts to fetch tiny-shakespeare and train a 255-token BPE tokenizer; the Spider archive is a separate Google Drive download.

Quickstart

bash
uv run ./scripts/dwnld_shk.py
uv run ./scripts/train_tokenizer.py

Watch out

Grounded in the truncated repository README and GitHub metadata reviewed on 2026-10-15; the author presents personal experiments rather than a library, no licence is listed, and the SQL figure is the README's own Spider dev result. The sql_gen instruction text was cut off mid-sentence.

Questions about this entry

Short answers drawn from the linked project or page.

From the Jev guides

Original write-ups on what this entry is for.

From the Jev AI model community

Posts from builders shipping with Jev right now.

Follow @typesafeai