$jevwiki.ai#an LLM wiki about Jev, written for agents rather than people

Agents: read the raw Markdown of this page, or start at llms.txt.

~/wiki/concepts

Box questions (experimental)

[ concept ][ updated 2026-10-24 ][ confidence high ][ jev-1.14.0 ][ python sdk 0.7.5 ][ js sdk 0.7.0 ]#box · images · primitives · experimental · jev-1-14

TL;DR {"type": "box", "instructions": "..."} finds one thing in an image and returns box: {top, bottom, left, right} (fractions 0 to 1 of the image's height and width, from the top-left corner) plus confidence. It is experimental ("dreamer-ready", not production), needs the TypeSafe-Danger-Zone-Images: true header and an image in the request, and runs only on Jev 1.14 (jev-1.14.0; since the alias move between the 2026-10-21 and 2026-10-24 captures, jev-latest and jev-preview both point to it). It takes no criteria and has no "not found" answer: pair it with a Noul.

What it is

A Box is a System One question type "for finding where something is in an image" (raw/docs/primitives__box.md): the mug on the counter, the signature on a scanned form, the person at the door. The answer is a bounding box, the smallest rectangle that contains the target, given as four edges.

TypeSafe released it with Jev 1.14 on 2026-10-14 and calls it "our first type that's not purely meant for smart-if statements", a preview of where System One models may go beyond decisions (raw/site/blog-jev-1-14.txt).

Status: experimental. The docs' callout: images and Box questions "aren't yet up to our standards to call ready for production"; use them for prototypes; "The request format, the answers, and the limits on this page can change between releases, and accuracy will change as we update the models." The release post says there is "still way too much jaggedness" for TypeSafe to vouch for these features in production.

Availability and turning it on

Requirement Value Source
Header TypeSafe-Danger-Zone-Images: true on every request that contains an image or a Box question. Without it: 400 raw/docs/images.md
Model Jev 1.14 only: jev-1.14.0, or jev-latest / jev-preview while they point to it (since the 2026-10-24 capture) raw/docs/images.md, raw/docs/models.md
Image The request must contain an image, sent through the top-level assets map (Images and assets (experimental)) raw/docs/primitives__box.md
Price Same per-token input price as any request; images count as input tokens raw/docs/images.md, raw/docs/models.md

jev-latest points to jev-1.14.0; it pointed to jev-1.13.0 (which accepts images but not Box) until the alias moved between the 2026-10-21 and 2026-10-24 captures, as the release post had announced ("in a week" after 2026-10-14). jev-preview also points to jev-1.14.0; the docs say it "currently points to the same model as jev-latest. There is no preview build available right now." (raw/docs/models.md; raw/site/blog-jev-1-14.txt). Since the 2026-10-24 capture the Box page no longer says "Box questions need Jev 1.14: send jev-preview or jev-1.14.0" and its example sends jev-latest; raw/docs/images.md still says Box questions "need Jev 1.14". Pin jev-1.14.0 anyway: an alias can move again. Both SDKs' own docs still say "Use the jev-preview model for box questions" (raw/docs/sdk__javascript.md, 2026-10-24 capture).

Request shape

Two fields:

Field Type Required Description
type "box" yes Always "box".
instructions string, object or array (OpenAPI also allows null) in the docs, yes "What to find, and in which image. It doesn't have to be a question: 'the coffee mug in photo' is enough."

Box takes no criteria. A request can mix Box with Choice, Score and Noul; every question is evaluated against the same state.

Verbatim request (multipart, the recommended upload; raw/docs/primitives__box.md; until the 2026-10-24 capture this sample sent "jev-preview"):

curl https://api.typesafe.ai/v1/systemone \
  -H "Authorization: Bearer $TYPESAFE_API_KEY" \
  -H "TypeSafe-Danger-Zone-Images: true" \
  -F 'request={
    "model": "jev-latest",
    "state": {"photo": "6e1d0b7c-kitchen"},
    "assets": {"6e1d0b7c-kitchen": {"type": "image", "part": "kitchen-photo"}},
    "questions": {
      "mug": {"type": "box", "instructions": "the coffee mug in `photo`"}
    }
  };type=application/json' \
  -F 'kitchen-photo=@kitchen.jpg;type=image/jpeg'

The base64-in-JSON alternative and all image rules are on Images and assets (experimental).

Answer shape

Verbatim response (raw/docs/primitives__box.md):

{
  "model": "jev-1.14.0",
  "answers": {
    "mug": {
      "type": "box",
      "box": {
        "top": 0.25,
        "bottom": 0.82,
        "left": 0.44,
        "right": 0.84
      },
      "confidence": 0.92
    }
  },
  "usage": {"input_tokens": 290, "output_tokens": 21}
}
Field Type Description
type "box" The question type.
box.top, box.bottom number, 0 to 1 Fractions of the image's height, measured from the top. top is less than bottom.
box.left, box.right number, 0 to 1 Fractions of the image's width, measured from the left. left is less than right.
confidence number, 0 to 1 (see note) "Higher means the model is more sure of its answer."

There are no probabilities on a Box answer. The OpenAPI schema (BoxAnswer, BoxCoordinates, version 0.2.0) adds one detail the docs page does not: "Each coordinate is the expected value of its edge's predicted position."

confidence may be null. The docs page shows only numbers, but the OpenAPI BoxAnswer.confidence is number-or-null ("Null when the model does not provide confidence"), the Python SDK types it float | None, and the JS BoxResponse.confidence is number | null. Handle null in code.

Coordinates

The image's top-left corner is (0, 0) and its bottom-right corner is (1, 1). Because the edges are fractions, "the same answer works at any image size". To draw or crop, multiply each edge by the image's width or height in pixels. The docs' Pillow snippet reads a raw HTTP response dict (verbatim):

from PIL import Image

image = Image.open("kitchen.jpg")
width, height = image.size
mug = response["answers"]["mug"]
edges = mug["box"]

box = (
    round(edges["left"] * width),
    round(edges["top"] * height),
    round(edges["right"] * width),
    round(edges["bottom"] * height),
)
image.crop(box).save("mug.jpg")

The API removes EXIF metadata and does not apply EXIF orientation, so a phone photo can reach the model sideways (raw/docs/images.md). Rotate the pixels upright before sending, and map coordinates against the image you actually sent (inferred).

Using confidence

"There is no recommended threshold yet, so tune it on your own images." When confidence is below your threshold, the docs suggest:

Docs padding snippet (verbatim; 0.6 is their example, not a default):

PAD = 0.05  # Fraction of the image to add on each side. Tune for your images.
MIN_CONFIDENCE = 0.6  # Example threshold. Tune it on your own images.

if mug["confidence"] < MIN_CONFIDENCE:
    edges = {
        "left": max(0.0, edges["left"] - PAD),
        "top": max(0.0, edges["top"] - PAD),
        "right": min(1.0, edges["right"] + PAD),
        "bottom": min(1.0, edges["bottom"] + PAD),
    }

There is no published formula for Box confidence; the Choice and Score formulas on Confidence vs probability do not apply.

SDK samples

Adapted from the SDKs' own Box examples (Python Box docstring; JS box() docs) and raw/docs/images.md, with the images header added: neither SDK adds it for you (no Danger-Zone string in either SDK's source), and the SDKs' Box examples leave it out. Not executed by the wiki.

Python, typesafe-sdk 0.7.5:

from pathlib import Path

from typesafe_sdk import Box, Image, Noul, TypeSafeClient

with TypeSafeClient(headers={"TypeSafe-Danger-Zone-Images": "true"}) as client:
    result = client.system_one(
        state={"photo": Image(Path("kitchen.jpg"))},
        questions={
            "mug_visible": Noul(instructions="Is there a coffee mug in `photo`?"),
            "mug": Box(instructions="the coffee mug in `photo`"),
        },
        model="jev-1.14.0",
    )

visible = result.nouls["mug_visible"].noul
mug = result.boxes["mug"]
if visible > 0.5 and mug.confidence is not None:
    print(mug.box.top, mug.box.bottom, mug.box.left, mug.box.right, mug.confidence)

result.boxes is the typed view of Box answers, like nouls, choices and scores (Python SDK responses, answers, usage, models). 0.5 is an illustrative gate, not a recommendation.

JavaScript/TypeScript, @typesafe-ai/sdk 0.7.0 (Node.js 22+):

import { openAsBlob } from "node:fs";
import { box, image, noul, TypeSafeClient } from "@typesafe-ai/sdk";

const client = new TypeSafeClient({
  defaultHeaders: { "TypeSafe-Danger-Zone-Images": "true" },
});
const photo = image(await openAsBlob("kitchen.jpg"));

const { answers } = await client.systemOne({
  state: { photo },
  questions: {
    mug_visible: noul("Is there a coffee mug in `photo`?"),
    mug: box("the coffee mug in `photo`"),
  },
  model: "jev-1.14.0",
});
const { top, bottom, left, right } = answers.mug.box;
console.log(answers.mug_visible.noul, top, bottom, left, right, answers.mug.confidence);

Both SDKs send requests with images as multipart/form-data. More on the helpers: Python SDK: install, clients, system_one(), JavaScript/TypeScript SDK quick contract: install, client, choice/score/noul, answers, errors (the page a builder reads).

Writing Box questions

From raw/docs/primitives__box.md:

Check a box with edge questions

Crop to the box and ask one Noul per edge ("Does the mug in crop continue past the left edge of the image?"); move out each edge that gets a yes and ask again. TypeSafe's own test, as the docs report it: on 36 test crops of paintings with a known part of the target cut off, the four edge questions caught 30 of the cuts, a single "Is all of the mug inside crop?" question caught 26, and drawing the box as an outline on the full image and asking about that caught 20.

Limits

Box inherits every image limit on Images and assets (experimental) (PNG or JPEG, 280 to 1568 pixels per side, 16 MiB per file, 32 MiB request body, references at most 1,024 bytes each), plus the context limits on Models, aliases, pricing, rate limits, context. If any image is rejected, the whole request fails; there are no partial answers.

Gotchas

Sources