---
title: "Images and assets (experimental)"
type: concept
tags: [images, assets, multimodal, experimental, state]
created: 2026-10-15
updated: 2026-10-24
confidence: high
sources:
  - raw/docs/images.md
  - raw/docs/primitives__box.md
  - raw/site/openapi.json
  - raw/docs/models.md
  - raw/docs/concepts__state.md
  - raw/docs/api__errors.md
  - raw/docs/api.md
  - raw/docs/sdk__python__usage__basic.md
  - raw/github/typesafe-sdk-python/src/typesafe_sdk/_core/assets.py
  - raw/github/typesafe-sdk-python/pyproject.toml
  - raw/docs/sdk__javascript.md
  - raw/docs/sdk__javascript__changelog.md
  - raw/site/blog-jev-1-14.txt
jev_version: "jev-1.14.0"
sdk_python: "0.7.5"
sdk_js: "0.7.0"
summary: "Experimental, header-gated images: list each PNG/JPEG once in a top-level assets map, write its reference key in state, instructions or criteria; works on Jev 1.13 and 1.14."
---

# Images and assets (experimental)

> **TL;DR** Since 2026-10-14 a request can carry images, as an **experimental** feature ("dreamer-ready", not production). Send `TypeSafe-Danger-Zone-Images: true`, list each image once in a top-level `assets` map keyed by a reference string, and write that string wherever the image belongs in `state`, `instructions` or `criteria`. PNG or JPEG; multipart upload recommended, base64 allowed. Each place a reference appears adds the image again toward the limits. Works on `jev-1.13.0` and `jev-1.14.0`; the `box` type needs 1.14, which `jev-latest` points to since the alias move between the 2026-10-21 and 2026-10-24 captures (until then it pointed to `jev-1.13.0`, so Box needed a pin or `jev-preview`).

## What it is

"An image can go anywhere text can go in a System One request: in the `state`, in a question's `instructions`, and in its `criteria`. Every question type accepts images" (raw/docs/images.md). The [[concepts/box]] type returns where something is in an image.

TypeSafe's release post: "the Jev you all know and love (`jev-1.13.0`) was multimodal all along", and "all models (Jev included) perform worse on multimodality over pure text. Breaking down questions into the simplest possible decompositions helps a lot" (raw/site/blog-jev-1-14.txt, .html for the model name).

**Status: experimental.** Images "aren't yet up to our standards to call ready for production"; "The request format, the answers, and the limits on this page can change between releases, and accuracy will change as we update the models" (raw/docs/images.md). Use them for prototypes.

## Turning it on

Images are off by default. Send `TypeSafe-Danger-Zone-Images: true` with **each** request that contains an image or a Box question. Without it, such a request returns `400`.

| Where | How |
|---|---|
| HTTP | `-H "TypeSafe-Danger-Zone-Images: true"` |
| Python `typesafe-sdk` 0.7.5 | On the client, `TypeSafeClient(headers={"TypeSafe-Danger-Zone-Images": "true"})`, or per call, `extra_headers={...}` |
| JS `@typesafe-ai/sdk` 0.7.0 | On the client, `new TypeSafeClient({ defaultHeaders: { "TypeSafe-Danger-Zone-Images": "true" } })`, or per call, the `headers` option |

Neither SDK sends the header for you (no `Danger-Zone` string in either SDK's source, checked 2026-10-15).

## The `assets` map and references

Each image is listed once in the request's top-level `assets` map. Its key is a **reference string**. Write the reference wherever the image belongs, as a whole string value or inside a longer string, and "the model receives the image in its place."

### Asset fields

| Field | Type | Required | Description |
|---|---|---|---|
| `type` | `"image"` | no | If sent, must be `"image"`. |
| `part` | string | one of `part` / `data_base64` | In a multipart request, the name of the file part holding the image. |
| `data_base64` | string | unless `part` is set | The file, base64-encoded: standard alphabet with `=` padding, no line breaks or whitespace, no `data:` prefix. |
| `format` | `"png"` \| `"jpg"` \| `"jpeg"` | with `data_base64`; optional with `part` | Lowercase. `"jpeg"` is another name for `"jpg"`. The OpenAPI adds: "The actual format is detected from the bytes." |

Send each image one way. If an asset has both `part` and `data_base64`, the API uses `part`. The OpenAPI `assets` map has `minProperties: 1` and recommends "a per-image UUID" as the key.

### Reference rules

References are matched as plain text anywhere in the request's strings.

- **Every asset must be referenced** somewhere in `state`, `instructions` or `criteria`; an unreferenced asset returns `400`.
- **Whole values match first.** A string that is exactly a reference becomes that image; in longer strings, references are replaced left to right, never overlapping.
- **Each use adds the image again.** Uploaded once, but every appearance adds another copy to what the model reads, and every copy counts toward the expanded-request limit.
- **Object keys are not replaced.** A Choice option name or question ID stays text; put a picture in the option's description instead.
- **Answers contain the reference, not the image**, wherever they repeat your criteria (e.g. a Score's `legend`).
- **Key size:** at most 1,024 bytes of UTF-8 per reference and 65,536 bytes for all references in a request together; bytes, not characters. Not empty, not only whitespace.
- **Avoid accidental matches.** A short reference such as `cat` could match inside `category`. "A new UUID for each image in each request is the safest choice."

Verbatim example (raw/docs/images.md; base64 truncated upstream):

```json
{
  "model": "jev-latest",
  "state": {
    "photo": "a3f1c2e8-found"
  },
  "assets": {
    "a3f1c2e8-found": {"type": "image", "format": "jpg", "data_base64": "/9j/4AAQSkZJRg..."},
    "77b0d4a1-poster": {"type": "image", "format": "jpg", "data_base64": "/9j/4AAQSkZJRg..."}
  },
  "questions": {
    "same_dog": {
      "type": "noul",
      "instructions": "Is the dog in `photo` the same dog as in this picture: 77b0d4a1-poster?"
    },
    "has_collar": {
      "type": "noul",
      "instructions": "Is the dog in `photo` wearing a collar?"
    },
    "coat": {
      "type": "choice",
      "instructions": "What is the coat of the dog in `photo` like?",
      "criteria": {
        "short": "Short and smooth.",
        "long": "Long or feathered.",
        "curly": "Curly or wiry."
      }
    }
  }
}
```

## Where the image goes

| Placement | Behaviour | Use for |
|---|---|---|
| `state` | Part of the state; every question is evaluated against it. Mix with text, several images, nested or labelled; point at one by path (`` `frames[2].frame` ``). | Images several questions use |
| `instructions` | Part of that one question; sent once **per question that includes it**. | An image only one question needs (e.g. a known badge photo to compare against) |
| `criteria` | Choice options, Score levels, Noul `true`/`false` can each include pictures, alone or as `{"description": ..., "examples": [...]}` (key names are yours). | Defining options by example |

When several questions use one image, put it in `state` once and refer to it by path: fewer tokens and less of the size limits. References work at any level of nested JSON ([[concepts/advanced-structure]]).

Upstream's pattern table: compare against a known picture (known picture in `instructions`, photo in `state`); define options by example (`criteria`); what changed across frames (`state`); look closer only when needed (small copy first, then a larger copy or crop); search a large image by tiles as Choice options (`criteria`); find where it is with [[concepts/box]].

## Uploading over HTTP

**Multipart is recommended**: it sends the file as is, while base64 makes each image about a third larger, and that counts toward the 32 MiB body limit. Verbatim (raw/docs/images.md):

```bash
curl https://api.typesafe.ai/v1/systemone \
  -H "Authorization: Bearer $TYPESAFE_API_KEY" \
  -H "TypeSafe-Danger-Zone-Images: true" \
  -F 'request={
    "model": "jev-latest",
    "state": {"door": "a3f1c2e8-door"},
    "assets": {"a3f1c2e8-door": {"type": "image", "part": "door-photo"}},
    "questions": {
      "person_present": {"type": "noul", "instructions": "Is there a person in `door`?"}
    }
  };type=application/json' \
  -F 'door-photo=@door-camera.jpg;type=image/jpeg'
```

Multipart rules: the first part is named `request` and holds the JSON body (`application/json`); part names are unique and non-empty, `request` is reserved; each image part has content type `image/png`, `image/jpeg`, `image/jpg`, `application/octet-stream`, or none (anything else: `400`); each file part belongs to exactly one asset (an orphan part, two assets naming one part, or a missing part: `400`). To reuse an image, write its reference in each place.

Base64: put the encoded file in `data_base64` and set `format`; Python's `base64.b64encode` gives the right encoding. The docs give a full `requests` example; the shape is the JSON example above.

## SDK helpers

Verbatim from raw/docs/images.md (Python 0.7.5 and JS 0.7.0 SDKs):

```python
from pathlib import Path

from typesafe_sdk import Choice, Image, Noul, TypeSafeClient

with TypeSafeClient() as client:
    result = client.system_one(
        state={"door": Image(Path("door-camera.jpg"))},
        questions={
            "person_present": Noul(instructions="Is there a person in `door`?"),
            "carrying": Choice(
                instructions="What is the person in `door` carrying?",
                criteria={
                    "package": "A box or parcel.",
                    "bag": "A bag of groceries or shopping.",
                    "nothing": "Empty hands.",
                },
            ),
        },
        extra_headers={"TypeSafe-Danger-Zone-Images": "true"},
    )
    print(result.answers)
```

```ts
import { openAsBlob } from "node:fs";
import { choice, image, noul, TypeSafeClient } from "@typesafe-ai/sdk";

const client = new TypeSafeClient();
const door = image(await openAsBlob("door-camera.jpg"));

const { answers } = await client.systemOne(
  {
    state: { door },
    questions: {
      person_present: noul("Is there a person in `door`?"),
      carrying: choice("What is the person in `door` carrying?", {
        package: "A box or parcel.",
        bag: "A bag of groceries or shopping.",
        nothing: "Empty hands.",
      }),
    },
  },
  { headers: { "TypeSafe-Danger-Zone-Images": "true" } },
);
console.log(answers);
```

The SDKs create references and upload images as `multipart/form-data` for you.

| | Python `typesafe-sdk` 0.7.5 | JS `@typesafe-ai/sdk` 0.7.0 |
|---|---|---|
| Create an image | `Image(Path(...))`, `Image(bytes or any buffer)`, `Image.from_base64(...)`, `Image.from_pil(img, format="jpeg")` (needs the `pil` extra, `pillow>=11.0.0`) | `image(bytes or Blob/File)`, `Image.fromBase64(...)` |
| Place directly in content | Put the `Image` in `state`, instructions or criteria | Put the `image()` value in `state`, instructions or criteria |
| Inside text | `with client.attach_assets(photo):` then `f"... {photo} ..."` (`str(image)` is its reference; without the `with` block the reference is sent as plain text) | `` withAssets`... ${upload} ...` `` and `withAssets.join(list, ", ")`; an image in an *untagged* template literal rejects the call with `TypeSafeError` |
| Your own references | `assets={"<product>": Image(photo)}` | `assets: { "<delivered>": image(...) }`; every name must appear in the request or the call rejects before sending |

Details: [[reference/python-sdk]], [[reference/javascript-sdk]].

## Limits

From raw/docs/images.md; all limits include both ends.

| Limit | Value |
|---|---|
| Format | PNG or JPEG, still images only. Animated PNGs are rejected. |
| File size | 16 MiB per image file (after decoding, for base64). |
| Width and height | 280 to 1568 pixels each; an image with either side under 280 pixels is rejected (the docs said 256 until 2026-10-15). |
| Total pixels | 78,400 to 2,458,624 (280 × 280 to 1568 × 1568). |
| Request body | 32 MiB including JSON, base64 and multipart framing; headers don't count; compression doesn't raise it. |
| All images in the request | 32 MiB for every distinct image after processing; re-encoding can make an image larger than the file you sent. |
| Expanded request | 32 MiB after every reference is replaced by its image; each use adds the image again (a 10 MiB processed image referenced four times is over). |
| Reference keys | 1,024 bytes each; 65,536 bytes in total. |

An image outside the limits is rejected, never resized. If any image is rejected, the whole request fails; no partial answers. Images have no separate rate limits. How many images fit also depends on the context length ([[reference/models-and-pricing]]; the experimental long-context header raises it, see [[concepts/state]]).

## Tokens and price

"Each image counts as input tokens, billed at the same per-token input price as text. There is no separate charge for images." `usage.input_tokens` includes every image, and image tokens count toward the context length. As of 2026-10-15 TypeSafe says it will provide "guidance on estimating image token usage soon"; there is no published per-image token formula.

## Preparing images

- **Resize** so the longest side is at most 1568 pixels (a 12-megapixel phone photo becomes 1568 × 1176); scale up or pad below the minimum.
- **Rotate the pixels upright.** The API strips all metadata (EXIF, colour profiles, comments) and does **not** apply EXIF orientation.
- **Still images only**: export a frame from animation or video; convert GIF, WebP or HEIC to PNG or JPEG.
- **Prefer a standard 8-bit baseline JPEG**; re-save progressive JPEGs. Re-export a file the API rejects.

Docs snippet (verbatim):

```python
from PIL import Image, ImageOps

with Image.open("photo.jpg") as original:
    img = ImageOps.exif_transpose(original)  # rotate the pixels upright
    img.thumbnail((1568, 1568))  # longest side at most 1568, same aspect ratio
    # thumbnail only shrinks: upscale or pad an image under the minimum separately
    img.convert("RGB").save("photo-ready.jpg", "JPEG", quality=90)  # baseline JPEG
```

## Errors

| Status | Cause |
|---|---|
| `400` | Image or Box question without the `TypeSafe-Danger-Zone-Images: true` header. |
| `400` | Asset rejected: not PNG/JPEG, damaged, animated, below the minimum size, unreferenced, bad base64 or `format`, empty reference, or a broken multipart rule. |
| `400` | `max_tokens_exceeded`: images plus text exceed the context length. Send fewer or smaller images, or move them from `instructions` into `state`. |
| `413` | Over a size limit: image file, dimensions or pixels, request body, all images, expanded request, or reference lengths. |

If a request has several problems, the error reports one. General status codes: [[reference/rate-limits-and-errors]].

## Docs inconsistencies (as of 2026-10-15)

- raw/docs/models.md's model table lists **Input: "Text"** for both Jev 1.14 and Jev 1.13, and its Input bullet says Jev "reads natural-language text", while raw/docs/images.md says "Jev 1.13 and Jev 1.14 both accept images". The models page also dropped images from its "does not accept" list (it now names audio and video only). Read the table as the default, non-experimental input (inferred); the images page is the contract for the experimental feature.
- raw/docs/concepts__state.md now says "Audio and video are not supported (yet)" and no longer mentions images, without saying that images are header-gated ([[concepts/state]]).
- raw/docs/api.md's prose describes neither `assets` nor `box`; raw/site/openapi.json (version 0.2.0) defines `Asset`, the `assets` map, `BoxQuestion`, `BoxAnswer` and `BoxCoordinates`. Its `assets` description adds "Image assets require a vision-language model", which the docs do not explain.
- raw/docs/sdk__python__usage__basic.md says image support is dreamer-ready "As of November 14, 2026", a date after this capture; the feature shipped 2026-10-14 (raw/docs/sdk__python__changelog.md v0.7.5).

## Related

- [[concepts/primitives]] — every question type accepts images
- [[concepts/box]] — locating a target in an image (Jev 1.14)
- [[concepts/state]] — state shapes, limits, long context
- [[concepts/advanced-structure]] — references inside structured fields
- [[reference/http-api]] — the wire contract
- [[reference/openapi-schemas]] — `Asset`, `SystemOneRequest.assets`
- [[reference/models-and-pricing]] — models, price, context length
- [[reference/python-sdk]], [[reference/javascript-sdk]] — `Image`, `image()`, `attach_assets`, `withAssets`

## Sources

- raw/docs/images.md (https://docs.typesafe.ai/images)
- raw/docs/primitives__box.md (https://docs.typesafe.ai/primitives/box)
- raw/site/openapi.json (https://api.typesafe.ai/openapi.json) — `Asset`, `assets`
- raw/docs/models.md (https://docs.typesafe.ai/models) — Input row, context length
- raw/docs/concepts__state.md (https://docs.typesafe.ai/concepts/state)
- raw/docs/api__errors.md (https://docs.typesafe.ai/api/errors) — request rules, compression, status codes, rate-limit handling (since 2026-10-20; the same text was on raw/docs/api.md before)
- raw/docs/api.md (https://docs.typesafe.ai/api) — request rules
- raw/docs/sdk__python__usage__basic.md (https://docs.typesafe.ai/sdk/python/usage/basic) — attaching assets
- raw/github/typesafe-sdk-python/src/typesafe_sdk/_core/assets.py — `Image` constructors
- raw/github/typesafe-sdk-python/pyproject.toml — `pil` extra
- raw/docs/sdk__javascript.md (https://docs.typesafe.ai/sdk/javascript) — `image()`, `withAssets`, `assets`
- raw/docs/sdk__javascript__changelog.md (https://docs.typesafe.ai/sdk/javascript/changelog) — v0.7.0
- raw/site/blog-jev-1-14.txt, raw/site/blog-jev-1-14.html (https://typesafe.ai/blog/jev-1-14) — release post