A box question returns one bounding box as four fractions of the image, plus a confidence. It is the most spatial primitive Jev exposes, and the easiest one to misuse. People ask it to find a value in a scanned invoice, then treat the four edges as if they came with a guarantee. The box docs are explicit about the limits: one box per question, no criteria field, no not-found value, and the whole feature is experimental.
Used well, box questions are a useful grounding layer for documents, charts, and interfaces. Used as an OCR replacement, they are a confident source of wrong crops.
What a Box answer is
The docs define the answer precisely: top, bottom, left, and right, each between 0 and 1 as a fraction of the image, plus a confidence. The same answer works at any image size because the coordinates are relative. Reading it is arithmetic: multiply each edge by the image width or height, then crop, pad, or re-ask at higher resolution.
Box takes no criteria. The instructions describe what to find and in which image, and may be a phrase like “the coffee mug in photo.” There is no none value. An absent target still returns four edges, which is the failure mode to design around.
The feature also has a version gate. Box needs jev-1.14.0 or jev-preview, plus the TypeSafe-Danger-Zone-Images: true header. jev-1.13.0 returns HTTP 400 for a Box question. The Jev 1.14 article covers the release context.
Pair Box with a presence check
Because Box cannot say “not found,” the docs recommend pairing it with a Noul that checks whether the target is present. That is not a formality. A box on an absent target is a crop of whatever the model considered closest, and a downstream step may read it as if it were the thing you asked for.
The pattern is two questions over the same image asset:
- A Noul: “Is the shipping label visible in
image_1?” - A Box: “Return the bounding box of the shipping label in
image_1.”
If the Noul is below your threshold, skip the crop. If it is high, the box is still a candidate until you verify the crop. For high-stakes documents, add a third step: ask a text question about the cropped region and check that it matches the expected field type.
The confidence notes are the reference for the threshold side. The box's own confidence is useful, but it is not a presence probability.
Documents are not screenshots
A screenshot has a stable layout and a small number of targets. A scanned invoice has tables, stamps, handwriting, and fields that move between vendors. A chart has axes, legends, and data labels that may be too small to resolve at the image's native size.
The image assets docs set the practical limits: PNG or JPEG still images only, 256 to 1568 pixels per side, up to 16 MiB per file, and 32 MiB per request body after processing. They also say all models perform worse on images than on text and recommend decomposing questions into the simplest possible form.
For documents, that usually means a pipeline: parse the text first, use box questions only for the regions where text is ambiguous or missing, and re-ask at higher resolution when confidence is low. The pre-parsed extraction cookbook is the text-side template: find candidates with code, ask the model to pick, and copy the span verbatim. The structure recovery cookbook is the layout-side template: stitch lines, then classify blocks.
Charts and interfaces need a different question
A chart is not a document with text. The target might be a bar, a point, or a legend entry, and the correct box depends on what you plan to do with it. Ask for the smallest region that contains the evidence you need, not the whole chart.
For interfaces, the same rule applies. A button has a hit area, a label, and an icon; the box you want depends on whether you are clicking, reading, or verifying. The browser and computer-use article covers action gating; the box question is the grounding step that tells the agent where the target is before it acts.
Two habits help:
- Ask for one target per question. Multiple targets in one instruction make the box ambiguous.
- Ask for the region at the resolution you will use. A box around a 12-pixel label is not useful if the crop is then downsampled.
Verify before you act
A box is a proposal, not a fact. Before an irreversible action, verify the crop. For a document, read the cropped field and check it against the expected type. For an interface, confirm the element is the one you intended before clicking. For a chart, check that the box contains the data label you need.
The jaggedness notes are the general checklist. Box does not escape the literal-reading problem: if the instruction says “the total,” the model may box the subtotal because that is what it sees. Use a specific instruction and a verification question.
A short checklist
- Gate Box behind
jev-1.14.0orjev-previewand the images header. - Pair every Box with a presence Noul; skip the crop when presence is low.
- Treat the box as a proposal; verify the crop before acting.
- Keep images within the documented size and payload limits.
- Use text extraction first when the document has a text layer.
- Ask for one target per question and at the resolution you will use.
- Re-ask at higher resolution when the box confidence is low.
Box questions are a grounding tool, not an oracle. The value is in the pipeline around them: presence, crop, verify, then act.