---
title: "Box questions (experimental)"
type: concept
tags: [box, images, primitives, experimental, jev-1-14]
created: 2026-10-15
updated: 2026-10-24
confidence: high
sources:
  - raw/docs/primitives__box.md
  - raw/docs/images.md
  - raw/site/openapi.json
  - raw/docs/api__system-one.md
  - raw/docs/sdk__python__api__types__questions.md
  - raw/docs/sdk__python__api__types__responses.md
  - raw/docs/sdk__javascript.md
  - raw/docs/sdk__javascript__api__functions__box.md
  - raw/docs/sdk__javascript__api__interfaces__BoxResponse.md
  - raw/github/typesafe-sdk-python/src/typesafe_sdk/_core/client/sync/client.py
  - raw/docs/models.md
  - raw/site/blog-jev-1-14.txt
jev_version: "jev-1.14.0"
sdk_python: "0.7.5"
sdk_js: "0.7.0"
summary: "Experimental Box type (Jev 1.14, now behind jev-latest; images header): locate one target in an image; answer is top/bottom/left/right as 0-1 fractions plus confidence; no not-found answer."
---

# Box questions (experimental)

> **TL;DR** `{"type": "box", "instructions": "..."}` finds **one** thing in an image and returns `box: {top, bottom, left, right}` (fractions 0 to 1 of the image's height and width, from the top-left corner) plus `confidence`. It is **experimental** ("dreamer-ready", not production), needs the `TypeSafe-Danger-Zone-Images: true` header and an image in the request, and runs only on Jev 1.14 (`jev-1.14.0`; since the alias move between the 2026-10-21 and 2026-10-24 captures, `jev-latest` and `jev-preview` both point to it). It takes no `criteria` and has no "not found" answer: pair it with a Noul.

## What it is

A Box is a System One question type "for finding where something is in an image" (raw/docs/primitives__box.md): the mug on the counter, the signature on a scanned form, the person at the door. The answer is a bounding box, the smallest rectangle that contains the target, given as four edges.

TypeSafe released it with Jev 1.14 on 2026-10-14 and calls it "our first type that's not purely meant for smart-if statements", a preview of where System One models may go beyond decisions (raw/site/blog-jev-1-14.txt).

**Status: experimental.** The docs' callout: images and Box questions "aren't yet up to our standards to call ready for production"; use them for prototypes; "The request format, the answers, and the limits on this page can change between releases, and accuracy will change as we update the models." The release post says there is "still way too much jaggedness" for TypeSafe to vouch for these features in production.

## Availability and turning it on

| Requirement | Value | Source |
|---|---|---|
| Header | `TypeSafe-Danger-Zone-Images: true` on every request that contains an image or a Box question. Without it: `400` | raw/docs/images.md |
| Model | Jev 1.14 only: `jev-1.14.0`, or `jev-latest` / `jev-preview` while they point to it (since the 2026-10-24 capture) | raw/docs/images.md, raw/docs/models.md |
| Image | The request must contain an image, sent through the top-level `assets` map ([[concepts/images]]) | raw/docs/primitives__box.md |
| Price | Same per-token input price as any request; images count as input tokens | raw/docs/images.md, raw/docs/models.md |

`jev-latest` points to `jev-1.14.0`; it pointed to `jev-1.13.0` (which accepts images but **not** Box) until the alias moved between the 2026-10-21 and 2026-10-24 captures, as the release post had announced ("in a week" after 2026-10-14). `jev-preview` also points to `jev-1.14.0`; the docs say it "currently points to the same model as `jev-latest`. There is no preview build available right now." (raw/docs/models.md; raw/site/blog-jev-1-14.txt). Since the 2026-10-24 capture the Box page no longer says "Box questions need Jev 1.14: send `jev-preview` or `jev-1.14.0`" and its example sends `jev-latest`; raw/docs/images.md still says Box questions "need Jev 1.14". Pin `jev-1.14.0` anyway: an alias can move again. Both SDKs' own docs still say "Use the `jev-preview` model for `box` questions" (raw/docs/sdk__javascript.md, 2026-10-24 capture).

## Request shape

Two fields:

| Field | Type | Required | Description |
|---|---|---|---|
| `type` | `"box"` | yes | Always `"box"`. |
| `instructions` | string, object or array (OpenAPI also allows `null`) | in the docs, yes | "What to find, and in which image. It doesn't have to be a question: 'the coffee mug in `photo`' is enough." |

Box takes **no `criteria`**. A request can mix Box with Choice, Score and Noul; every question is evaluated against the same state.

Verbatim request (multipart, the recommended upload; raw/docs/primitives__box.md; until the 2026-10-24 capture this sample sent `"jev-preview"`):

```bash
curl https://api.typesafe.ai/v1/systemone \
  -H "Authorization: Bearer $TYPESAFE_API_KEY" \
  -H "TypeSafe-Danger-Zone-Images: true" \
  -F 'request={
    "model": "jev-latest",
    "state": {"photo": "6e1d0b7c-kitchen"},
    "assets": {"6e1d0b7c-kitchen": {"type": "image", "part": "kitchen-photo"}},
    "questions": {
      "mug": {"type": "box", "instructions": "the coffee mug in `photo`"}
    }
  };type=application/json' \
  -F 'kitchen-photo=@kitchen.jpg;type=image/jpeg'
```

The base64-in-JSON alternative and all image rules are on [[concepts/images]].

## Answer shape

Verbatim response (raw/docs/primitives__box.md):

```json
{
  "model": "jev-1.14.0",
  "answers": {
    "mug": {
      "type": "box",
      "box": {
        "top": 0.25,
        "bottom": 0.82,
        "left": 0.44,
        "right": 0.84
      },
      "confidence": 0.92
    }
  },
  "usage": {"input_tokens": 290, "output_tokens": 21}
}
```

| Field | Type | Description |
|---|---|---|
| `type` | `"box"` | The question type. |
| `box.top`, `box.bottom` | number, 0 to 1 | Fractions of the image's **height**, measured from the top. `top` is less than `bottom`. |
| `box.left`, `box.right` | number, 0 to 1 | Fractions of the image's **width**, measured from the left. `left` is less than `right`. |
| `confidence` | number, 0 to 1 (see note) | "Higher means the model is more sure of its answer." |

There are no `probabilities` on a Box answer. The OpenAPI schema (`BoxAnswer`, `BoxCoordinates`, version 0.2.0) adds one detail the docs page does not: "Each coordinate is the expected value of its edge's predicted position."

**`confidence` may be `null`.** The docs page shows only numbers, but the OpenAPI `BoxAnswer.confidence` is number-or-null ("Null when the model does not provide confidence"), the Python SDK types it `float | None`, and the JS `BoxResponse.confidence` is `number | null`. Handle `null` in code.

## Coordinates

The image's top-left corner is (0, 0) and its bottom-right corner is (1, 1). Because the edges are fractions, "the same answer works at any image size". To draw or crop, multiply each edge by the image's width or height in pixels. The docs' Pillow snippet reads a raw HTTP response dict (verbatim):

```python
from PIL import Image

image = Image.open("kitchen.jpg")
width, height = image.size
mug = response["answers"]["mug"]
edges = mug["box"]

box = (
    round(edges["left"] * width),
    round(edges["top"] * height),
    round(edges["right"] * width),
    round(edges["bottom"] * height),
)
image.crop(box).save("mug.jpg")
```

The API removes EXIF metadata and does not apply EXIF orientation, so a phone photo can reach the model sideways (raw/docs/images.md). Rotate the pixels upright before sending, and map coordinates against the image you actually sent (inferred).

## Using confidence

"There is no recommended threshold yet, so tune it on your own images." When confidence is below your threshold, the docs suggest:

- **Ask again on a better image**: a larger version, or a higher-resolution crop around the box.
- **Pad the box before cropping** when the crop must not cut off any of the target.
- **Check the box with edge questions** (below).

Docs padding snippet (verbatim; `0.6` is their example, not a default):

```python
PAD = 0.05  # Fraction of the image to add on each side. Tune for your images.
MIN_CONFIDENCE = 0.6  # Example threshold. Tune it on your own images.

if mug["confidence"] < MIN_CONFIDENCE:
    edges = {
        "left": max(0.0, edges["left"] - PAD),
        "top": max(0.0, edges["top"] - PAD),
        "right": min(1.0, edges["right"] + PAD),
        "bottom": min(1.0, edges["bottom"] + PAD),
    }
```

There is no published formula for Box `confidence`; the Choice and Score formulas on [[concepts/confidence]] do not apply.

## SDK samples

Adapted from the SDKs' own Box examples (Python `Box` docstring; JS `box()` docs) and raw/docs/images.md, with the images header added: neither SDK adds it for you (no `Danger-Zone` string in either SDK's source), and the SDKs' Box examples leave it out. Not executed by the wiki.

Python, `typesafe-sdk` 0.7.5:

```python
from pathlib import Path

from typesafe_sdk import Box, Image, Noul, TypeSafeClient

with TypeSafeClient(headers={"TypeSafe-Danger-Zone-Images": "true"}) as client:
    result = client.system_one(
        state={"photo": Image(Path("kitchen.jpg"))},
        questions={
            "mug_visible": Noul(instructions="Is there a coffee mug in `photo`?"),
            "mug": Box(instructions="the coffee mug in `photo`"),
        },
        model="jev-1.14.0",
    )

visible = result.nouls["mug_visible"].noul
mug = result.boxes["mug"]
if visible > 0.5 and mug.confidence is not None:
    print(mug.box.top, mug.box.bottom, mug.box.left, mug.box.right, mug.confidence)
```

`result.boxes` is the typed view of Box answers, like `nouls`, `choices` and `scores` ([[reference/python-sdk-responses]]). `0.5` is an illustrative gate, not a recommendation.

JavaScript/TypeScript, `@typesafe-ai/sdk` 0.7.0 (Node.js 22+):

```ts
import { openAsBlob } from "node:fs";
import { box, image, noul, TypeSafeClient } from "@typesafe-ai/sdk";

const client = new TypeSafeClient({
  defaultHeaders: { "TypeSafe-Danger-Zone-Images": "true" },
});
const photo = image(await openAsBlob("kitchen.jpg"));

const { answers } = await client.systemOne({
  state: { photo },
  questions: {
    mug_visible: noul("Is there a coffee mug in `photo`?"),
    mug: box("the coffee mug in `photo`"),
  },
  model: "jev-1.14.0",
});
const { top, bottom, left, right } = answers.mug.box;
console.log(answers.mug_visible.noul, top, bottom, left, right, answers.mug.confidence);
```

Both SDKs send requests with images as `multipart/form-data`. More on the helpers: [[reference/python-sdk]], [[reference/javascript-sdk]].

## Writing Box questions

From raw/docs/primitives__box.md:

- **Name the target and the image.** With several images, the backticked path says which one to search.
- **The target can be a picture.** Put its reference in the instructions: "the person in `photo` who matches this badge photo: 9a21c7d3-badge".
- **A Box answer is always a box.** "There is no 'not found' answer, so a Box question about something absent still returns four edges." Ask a [[concepts/noul|Noul]] in the same request ("Is there a coffee mug in `photo`?") and ignore the box when it is low.
- **One box per question.** For several different things, one Box each. When the image may hold several of the same thing, describe the one you want ("the mug nearest the sink in `photo`"); when words can't tell them apart, add a Choice with `none`, `one` and `several`, and on `several` split the image into regions and ask again on each.
- **Crop large images before looking for small things.** "A target a few dozen pixels wide in a very large image can be missed." Find the region first (for example a Choice whose options are image tiles), Box on that tile, then convert its edges back to fractions of the full image.

### Check a box with edge questions

Crop to the box and ask one Noul per edge ("Does the mug in `crop` continue past the left edge of the image?"); move out each edge that gets a yes and ask again. TypeSafe's own test, as the docs report it: on 36 test crops of paintings with a known part of the target cut off, the four edge questions caught 30 of the cuts, a single "Is all of the mug inside `crop`?" question caught 26, and drawing the box as an outline on the full image and asking about that caught 20.

## Limits

Box inherits every image limit on [[concepts/images]] (PNG or JPEG, 280 to 1568 pixels per side, 16 MiB per file, 32 MiB request body, references at most 1,024 bytes each), plus the context limits on [[reference/models-and-pricing]]. If any image is rejected, the whole request fails; there are no partial answers.

## Gotchas

- **Experimental.** Format, answers and limits can change between releases. Do not build a production decision on it.
- **Wrong model.** Box needs Jev 1.14; `jev-1.13.0` does not run Box questions. `jev-latest` did not either until it moved to `jev-1.14.0` (between the 2026-10-21 and 2026-10-24 captures); code pinned to `jev-1.13.0` still cannot use Box.
- **Forgotten header.** Without `TypeSafe-Danger-Zone-Images: true` the request returns `400`, including SDK calls.
- **Absent target.** Four edges come back anyway; gate with a Noul.
- **`null` confidence.** Allowed by the schema and both SDK types.
- **Docs gaps (as of 2026-10-15).** raw/docs/api.md's prose did not mention `box` or `assets`, though raw/site/openapi.json defines `BoxQuestion`, `BoxAnswer`, `BoxCoordinates` and `Asset`; since 2026-10-20 the docs' System One page is a render of that spec, so it now shows them (raw/docs/api__system-one.md). raw/docs/primitives.md's type table still lists only Choice, Score and Noul. The OpenAPI `BoxQuestion` requires only `type`, while the docs page lists `instructions` as one of its two fields; send `instructions`.

## Related

- [[concepts/primitives]] — all question types and how to batch them
- [[concepts/images]] — the `assets` map, references, formats and limits Box depends on
- [[concepts/noul]] — the companion "is it there at all?" question
- [[concepts/choice]] — counting (`none` / `one` / `several`) and tile search
- [[concepts/confidence]] — why Box confidence has no published formula
- [[guides/choosing-a-primitive]] — when Box fits
- [[reference/http-api]] — the wire contract
- [[reference/openapi-schemas]] — `BoxQuestion`, `BoxAnswer`, `BoxCoordinates`
- [[reference/models-and-pricing]] — Jev 1.14, aliases, context limits, price
- [[reference/python-sdk]], [[reference/javascript-sdk]] — SDK `Box` / `box()` and image helpers

## Sources

- raw/docs/primitives__box.md (https://docs.typesafe.ai/primitives/box)
- raw/docs/images.md (https://docs.typesafe.ai/images) — header, assets, limits, EXIF
- raw/site/openapi.json (https://api.typesafe.ai/openapi.json) — `BoxQuestion`, `BoxAnswer`, `BoxCoordinates`
- raw/docs/api__system-one.md (https://docs.typesafe.ai/api/system-one) — the docs' render of that spec, since 2026-10-20
- raw/docs/sdk__python__api__types__questions.md (https://docs.typesafe.ai/sdk/python/api/types/questions) — Python `Box`
- raw/docs/sdk__python__api__types__responses.md (https://docs.typesafe.ai/sdk/python/api/types/responses) — `BoxAnswer.confidence` nullable, `boxes`
- raw/github/typesafe-sdk-python/src/typesafe_sdk/_core/client/sync/client.py — `headers` and `model` parameters
- raw/docs/sdk__javascript.md (https://docs.typesafe.ai/sdk/javascript) — images header, `box`
- raw/docs/sdk__javascript__api__functions__box.md (https://docs.typesafe.ai/sdk/javascript/api/functions/box)
- raw/docs/sdk__javascript__api__interfaces__BoxResponse.md (https://docs.typesafe.ai/sdk/javascript/api/interfaces/BoxResponse)
- raw/docs/models.md (https://docs.typesafe.ai/models) — Jev 1.14 and aliases
- raw/site/blog-jev-1-14.txt (https://typesafe.ai/blog/jev-1-14) — release post