Box is Jev's first question type that answers where. That sounds like a small addition next to Choice and Noul, but it changes what a decision model can sit in front of: a crop becomes a decision, a presence check becomes a decision, and a visual review queue becomes something code can assemble without a generative model in the loop.
The shape of the answer
A Box answer is four edges — top, bottom, left, right — each a fraction from 0 to 1, measured from the top-left corner of the image, plus a confidence. Because the edges are fractions, the same answer is correct at any image size: multiply by the pixel width and height and crop.
edges = response["answers"]["mug"]["box"]
box = (
round(edges["left"] * width),
round(edges["top"] * height),
round(edges["right"] * width),
round(edges["bottom"] * height),
)
image.crop(box).save("mug.jpg")
That is the whole readout. No text to parse, no coordinates to rescale from a 0–1000 grid, no schema to validate beyond four numbers in range.
Box always answers
The failure mode is the important design fact: there is no not-found value. Ask for a signature that is not on the form and you still get four edges and a confidence. The documented pattern is to pair the Box with a Noul presence check on the same state — one request, two questions, one evaluated in parallel with the other — and let code drop boxes whose presence probability is low.
That pairing is also how you handle a second problem: a low-confidence box is not necessarily wrong, it is uncertain. TypeSafe does not publish a recommended threshold yet, so the docs tell you to tune one on your own images.
The cut-off problem
A box that is nearly right is still a bad crop. TypeSafe's own test on 36 painting crops, each cutting off part of the target, is a good map of the options:
| Strategy | Crops caught (of 36) |
|---|---|
| Four per-edge Noul questions | 30 |
| One "is all of it inside" Noul | 26 |
| Draw the box and ask about the outline | 20 |
Per-edge questions win because they localize the failure — you learn which side is cut. The docs offer a cheaper mitigation too: pad the box outward before cropping when a crop must not lose any part of the target, and re-ask on a higher-resolution crop when confidence is low.
Two patterns worth building
Crop and escalate. Use Box to find the region, crop it, and send the crop to a stronger model or a human queue. This is the classic confidence-gated routing shape applied to pixels, and it keeps the expensive model reading a small, well-framed image instead of a whole page.
Verify and highlight. In a review tool, Box turns "check that the signature is in the right place" into a comparison of fractions against a template region — code decides, the model only locates. Community work like jev-box-study explores the hallucination-detector version of this.
The limits to respect
- Box needs
jev-1.14.0orjev-preview; on 1.13.0 it returns HTTP 400 before any model runs. - Every request with a Box question needs
TypeSafe-Danger-Zone-Images: true. - Images are PNG or JPEG stills, 256–1568 pixels per side, 16 MiB per file, 32 MiB per request body.
- The feature is dreamer-ready: format, limits, and accuracy can change between releases. Prototype, do not deploy.
Test on your own images
There is no recommended confidence threshold, and the accuracy numbers in the launch post are TypeSafe's own. Build a small labelled set — crops with known targets and known cut-offs — and measure two things separately: how often the presence Noul is right, and how often the box is good enough to crop. Those are different failures, and only one of them is fixed by padding.
