---
title: "Cookbook: OCR confidence and bounding boxes"
type: cookbook
tags: [cookbook, images, box, noul, documents]
created: 2026-10-29
updated: 2026-11-03
confidence: high
sources:
  - raw/docs/cookbooks__ocr_confidence_bounding_boxes.md
  - raw/docs/cookbooks.md
  - raw/docs/images.md
  - raw/docs/primitives__box.md
  - raw/docs/primitives__noul.md
  - raw/docs/sdk__python__usage__basic.md
jev_version: "jev-1.14.0"
sdk_python: "0.7.5"   # the cookbook installs typesafe-sdk[pil]>=0.7.5 and cooksafe>=0.2.0,<0.3.0 and pins model jev-1.14.0
summary: "Check OCR'd invoice values: three image requests per invoice (Box + Noul on the page, Box on zoomed regions, Noul on crops); a field under 0.25 on either check goes to review. Experimental images."
---

# Cookbook: OCR confidence and bounding boxes

> **TL;DR** Take the values an OCR-plus-LLM extractor read from invoice photos and check each one with Jev in three requests per invoice: (1) the whole photo with a `Box` (where is the field?) and a `Noul` (is this value that field on the page? = the **page check**); (2) a `Box` per field on an enlarged region around the first box, for a tighter box; (3) a `Noul` per field on a crop of that box (does the crop show the value? = the **crop check**). A field scoring below 0.25 on **either** check goes to review. On two supplier invoices with three misread values, all three were flagged and no correct value was; neither check alone caught all three. Images and Box are experimental and need the `TypeSafe-Danger-Zone-Images: true` header and Jev 1.14.

## Goal

OCR on invoice photos is "mostly right", but checking every value by hand "takes as long as typing them in" (raw/docs/cookbooks__ocr_confidence_bounding_boxes.md). The cookbook makes Jev the checker: it boxes each extracted value on the photo, scores it twice, and sends only low scorers to a person, with the box drawn so the reviewer sees each value where it is printed.

Upstream lists it under the new "Image processing" section, level Intermediate (raw/docs/cookbooks.md; [[cookbooks/overview]]). With [[cookbooks/reading-scanned-documents]] and [[cookbooks/feeder-watch]], it is one of the first three official worked examples of images and [[concepts/box|Box]] (all added 2026-10-29).

**Experimental.** The docs call images and `Box` "dreamer-ready": "not yet ready for production", and "their request format and answers can change between releases". They need Jev 1.14, and every request needs the `TypeSafe-Danger-Zone-Images: true` header; the cookbook's client sets it ([[concepts/images]]).

When to use it: you already extract fields from scans or photos (Tesseract, Apple Vision, an LLM) and want a second opinion that points a person at the likely misreads. When not to: as the extractor itself (the cookbook starts from the extracted values; Jev never reads the text out), or as an unreviewed production gate while images are experimental.

## Inputs / state shape

- **Two phone photos** of supplier invoices a restaurant received: a creased carbon copy from Milkwood Mushrooms (1393 × 1080 px) and a Siyakhula Trading invoice with "a shadow down the middle" (899 × 1599 px).
- **`extractions.json`**: the values an extractor produced, keyed by photo file name. Upstream used Apple Vision's text recognition and an LLM to structure its output; the intro says Tesseract and "your LLM of choice" also work, because the check needs only the photo and the values. Milkwood has 7 fields, Siyakhula 8 (`branch_code` too).
- **`FIELDS`**: one plain description per field, reused in every question, plus `SAME_VALUE`, an equivalence rule appended to the yes/no questions.

The `state` differs by request: request 1 sends `{"page": <the photo>}`; requests 2 and 3 send only a one-line text `task`, and every image goes inside its own question's `instructions`.

Numbers below came from `jev-1.14.0` on 2026-10-15 (the cookbook pins `TYPESAFE_MODEL = "jev-1.14.0"`; since the 2026-10-24 alias move `jev-latest` is also `jev-1.14.0`, [[reference/models-and-pricing]]). Every response is cached in a local `json_cache.json`, so a re-run reuses it. Until the 2026-11-03 docs the cookbook also published that cache in typesafe-ai/typesafe-public-examples, so a run reproduced the published numbers without calling the API, and said to delete it to send every request again; both lines are gone.

```python
import io
import json
import os
import urllib.request
from pathlib import Path

from cooksafe import JsonCache
from IPython.display import display
from PIL import Image as PILImage
from PIL import ImageDraw, ImageFont
from typesafe_sdk import Box, Image, Noul, SystemOneResponse, TypeSafeClient

TYPESAFE_MODEL = "jev-1.14.0"  # Box questions need Jev 1.14
BAR = 0.25  # a check scoring below this flags the field
ZOOM = 3  # the zoomed region is three times the first box, and at least ZOOM_MIN of the page
ZOOM_MIN = (0.30, 0.15)  # width, height as fractions of the page
GROW = (0.25, 0.20)  # review margin around each box, as fractions of its width and height
MIN_SIDE, MAX_SIDE = 280, 1568  # the pixel range Jev accepts for each side of an image

client = TypeSafeClient(
    api_key=os.environ.get("TYPESAFE_API_KEY", "cache-only"),  # keyless kernels replay the cache
    base_url=os.environ.get("TYPESAFE_ENDPOINT"),
    headers={"TypeSafe-Danger-Zone-Images": "true"},
    timeout=120.0,
)
json_cache = JsonCache(Path("json_cache.json"))


EXAMPLES = (
    "https://raw.githubusercontent.com/typesafe-ai/typesafe-public-examples/main/"
    "cookbooks/ocr_confidence_bounding_boxes"
)


def download(name: str) -> Path:
    """Fetch one of this cookbook's files from typesafe-public-examples unless it is on disk."""
    path = Path(name)
    if not path.exists():
        path.parent.mkdir(parents=True, exist_ok=True)
        urllib.request.urlretrieve(f"{EXAMPLES}/{name}", path)
    return path


FIELDS = {
    # field: what it is, in the words every question uses
    "supplier_name": "the supplier's (seller's) business name",
    "invoice_number": "the invoice or document number",
    "invoice_date": "the invoice date",
    "amount_due": "the final total amount payable",
    "bank_name": "the name of the supplier's bank in the payment details",
    "account_holder": "the account holder name in the supplier's bank details",
    "account_number": "the supplier's bank account number",
    "branch_code": "the supplier's bank branch code",
}
SAME_VALUE = "Ignore differences in spacing, punctuation, currency symbols and letter case."
INPUTS = Path("inputs")  # the photos, keyed in extractions.json by file name
EXTRACTIONS = json.loads(download("extractions.json").read_text())
for photo in EXTRACTIONS:
    download(f"inputs/{photo}")
```

(Upstream verbatim, reordered into one block. Install: `pip install ipython pillow 'typesafe-sdk[pil]>=0.7.5' 'cooksafe>=0.2.0,<0.3.0'`, then set `TYPESAFE_API_KEY`. Since the 2026-11-03 docs the code fetches `extractions.json` and the two photos (into `inputs/`) from typesafe-public-examples itself unless they are on disk; until then you downloaded them, with `json_cache.json`, by hand.)

**Image sizing in code.** Jev accepts 280 to 1568 pixels per side ([[concepts/images]]). The cookbook's `as_image()` shrinks larger pictures with `thumbnail`, enlarges small crops (`upscale_to`) "so their text is readable", and pastes anything still under 280 px into the middle of a white canvas, returning the picture's `frame` on that canvas so a box drawn on the canvas can be mapped back to the crop. A `crop(photo, box, pad)` helper cuts a fractional box out of the photo. The API itself never resizes: an out-of-range image is rejected (raw/docs/images.md).

```python
def as_image(picture: PILImage.Image, upscale_to: int | None = None) -> tuple[Image, tuple]:
    picture = picture.convert("RGB")
    if upscale_to and max(picture.size) < upscale_to:
        scale = upscale_to / max(picture.size)
        picture = picture.resize(
            (round(picture.width * scale), round(picture.height * scale)), PILImage.LANCZOS
        )
    picture.thumbnail((MAX_SIDE, MAX_SIDE))
    frame = (0.0, 0.0, 1.0, 1.0)  # left, top, width, height of the picture on the canvas
    if min(picture.size) < MIN_SIDE:
        canvas = PILImage.new("RGB", (max(picture.width, MIN_SIDE), max(picture.height, MIN_SIDE)), "white")
        x, y = (canvas.width - picture.width) // 2, (canvas.height - picture.height) // 2
        canvas.paste(picture, (x, y))
        frame = (x / canvas.width, y / canvas.height, picture.width / canvas.width,
                 picture.height / canvas.height)
        picture = canvas
    buffer = io.BytesIO()
    picture.save(buffer, "JPEG", quality=90)
    return Image(buffer.getvalue()), frame


def crop(photo: str, box: dict, pad: float = 0.0) -> PILImage.Image:
    with PILImage.open(INPUTS / photo) as page:
        w, h = page.size
        return page.convert("RGB").crop((
            max(0, int((box["left"] - pad) * w)), max(0, int((box["top"] - pad) * h)),
            min(w, int((box["right"] + pad) * w)), min(h, int((box["bottom"] + pad) * h)),
        ))
```

(Upstream verbatim.)

## Questions asked

| Request | State | Per field | `type` | `instructions` (verbatim template) |
|---|---|---|---|---|
| 1 · the photo | `{"page": <photo>}` | `box_<field>` | `box` | `` {FIELDS[field]}, printed as "{value}" in `page` `` |
| 1 | same | `fits_<field>` | `noul` | `` Is "{value}" {FIELDS[field]} on `page`? {SAME_VALUE} `` |
| 2 · zoomed regions | `{"task": "Locate fields in zoomed-in parts of a supplier invoice."}` | `<field>` | `box` | `{FIELDS[field]}, printed as "{value}" in this image: {image}` |
| 3 · crops | `{"task": "Check values read from cropped parts of a supplier invoice."}` | `<field>` | `noul` | `Image: {image}\nDoes this cropped image clearly show {FIELDS[field]} as "{value}"? {SAME_VALUE} Answer no if it is cut off, unreadable or different.` |

(Upstream uses typographic quotes, “ ”, around `{value}`.) No question has `criteria`: Box takes none, and the Nouls use plain yes/no. The extracted value is in every Box question as a hint "about which text to box". The page check is request 1's Noul P(yes); the crop check is request 3's.

Request 1 (verbatim):

```python
def page_questions(values: dict) -> dict:
    questions = {}
    for field, value in sorted(values.items()):
        questions[f"box_{field}"] = Box(
            instructions=f"{FIELDS[field]}, printed as “{value}” in `page`"
        )
        questions[f"fits_{field}"] = Noul(
            instructions=f"Is “{value}” {FIELDS[field]} on `page`? {SAME_VALUE}"
        )
    return questions


def as_json(questions: dict) -> str:
    return json.dumps(
        {key: q.model_dump(mode="json", exclude_none=True) for key, q in questions.items()}
    )


@json_cache
def find_fields(photo: str, questions_json: str) -> dict:
    questions = json.loads(questions_json)
    fields = sorted(key.removeprefix("box_") for key in questions if key.startswith("box_"))
    with PILImage.open(INPUTS / photo) as page:
        image, _ = as_image(page)  # a whole page fills the size range, so no canvas
    response = client.system_one(
        state={"page": image}, questions=questions, model=TYPESAFE_MODEL
    )
    return {
        "boxes": {f: response.boxes[f"box_{f}"].box.model_dump() for f in fields},
        "fits": {f: response.nouls[f"fits_{f}"].noul for f in fields},
    }
```

Request 2 cuts a region around each first box (three times its size, at least 30% of the page wide and 15% high, "so the field stays inside it even if the first box was off"), enlarges it to 1200 px, and puts each region's image in its own question; `client.attach_assets(...)` sends them all in one request and resolves the image references inside the text ([[reference/python-sdk]]). Since the 2026-11-03 docs the questions are built with a literal `{image}` placeholder (`IMAGE`), and `ask_with_images()` swaps in each field's image reference (`str(images[field])`) just before sending, so the cached functions take the questions as JSON without image data; until then each question embedded its image when it was built, and the cached functions took the extracted values and boxes instead. The code then maps each new box from canvas to region to page and pads it by `GROW` (25% of its width, 20% of its height on each side) "so that cropping to it can't cut off part of the value" (verbatim):

```python
IMAGE = "{image}"  # stands for each question's own image until the request is sent
ZOOM_TASK = "Locate fields in zoomed-in parts of a supplier invoice."


def region_around(box: dict) -> dict:
    width = min(1.0, max(ZOOM_MIN[0], (box["right"] - box["left"]) * ZOOM))
    height = min(1.0, max(ZOOM_MIN[1], (box["bottom"] - box["top"]) * ZOOM))
    left = min(max(0.0, (box["left"] + box["right"] - width) / 2), 1 - width)
    top = min(max(0.0, (box["top"] + box["bottom"] - height) / 2), 1 - height)
    return {"left": left, "top": top, "right": left + width, "bottom": top + height}


def zoom_questions(values: dict) -> dict:
    return {
        field: Box(
            instructions=f"{FIELDS[field]}, printed as “{values[field]}” in this image: {IMAGE}"
        )
        for field in sorted(values)
    }


def ask_with_images(task: str, questions: dict, images: dict) -> SystemOneResponse:
    """Put each field's image into its question, and send all the questions in one request."""
    for field, question in questions.items():
        question["instructions"] = question["instructions"].replace(IMAGE, str(images[field]))
    with client.attach_assets(list(images.values())):  # resolves the references in the text
        return client.system_one(state={"task": task}, questions=questions, model=TYPESAFE_MODEL)


@json_cache
def zoom_fields(
    photo: str, regions_json: str, task: str, questions_json: str, grow: tuple
) -> dict:
    regions, images, frames = json.loads(regions_json), {}, {}
    for field, region in regions.items():
        images[field], frames[field] = as_image(crop(photo, region), upscale_to=1200)
    response = ask_with_images(task, json.loads(questions_json), images)
    tight = {}
    for field, region in regions.items():
        b, (fx, fy, fw, fh) = response.boxes[field].box, frames[field]
        # canvas -> region -> page, then add the review margin
        left, right = (b.left - fx) / fw, (b.right - fx) / fw
        top, bottom = (b.top - fy) / fh, (b.bottom - fy) / fh
        gx, gy = (right - left) * grow[0], (bottom - top) * grow[1]
        rw, rh = region["right"] - region["left"], region["bottom"] - region["top"]
        tight[field] = {
            "left": max(0.0, region["left"] + (left - gx) * rw),
            "top": max(0.0, region["top"] + (top - gy) * rh),
            "right": min(1.0, region["left"] + (right + gx) * rw),
            "bottom": min(1.0, region["top"] + (bottom + gy) * rh),
        }
    return {"boxes": tight}
```

Request 3 (verbatim):

```python
CROP_TASK = "Check values read from cropped parts of a supplier invoice."


def crop_questions(values: dict) -> dict:
    return {
        field: Noul(
            instructions=f"Image: {IMAGE}\nDoes this cropped image clearly show {FIELDS[field]}"
            f" as “{values[field]}”? {SAME_VALUE} Answer no if it is cut off, unreadable or"
            " different."
        )
        for field in sorted(values)
    }


@json_cache
def check_crops(photo: str, boxes_json: str, task: str, questions_json: str) -> dict:
    images = {
        field: as_image(crop(photo, box, pad=0.01), upscale_to=800)[0]
        for field, box in json.loads(boxes_json).items()
    }
    response = ask_with_images(task, json.loads(questions_json), images)
    return {
        "shows": {f: response.nouls[f].noul for f in sorted(images)},
    }
```

## Combining logic in code

Jev returns boxes and probabilities; the review rule is one line of code. A field goes to review when either check is below `BAR` (verbatim):

```python
def check_invoice(photo: str, values: dict) -> dict:
    first = find_fields(photo, as_json(page_questions(values)))
    regions = {field: region_around(box) for field, box in first["boxes"].items()}
    tight = zoom_fields(photo, json.dumps(regions, sort_keys=True), ZOOM_TASK,
                        as_json(zoom_questions(values)), GROW)
    shown = check_crops(photo, json.dumps(tight["boxes"], sort_keys=True), CROP_TASK,
                        as_json(crop_questions(values)))
    return {
        field: {
            "value": values[field],
            "box": tight["boxes"][field],
            "fits": first["fits"][field],
            "shows": shown["shows"][field],
            "review": first["fits"][field] < BAR or shown["shows"][field] < BAR,
        }
        for field in sorted(values)
    }
```

The Box `confidence` is not used in the rule: the first box only decides where to zoom, and the tight box only decides the crop and the outline a reviewer sees. An `annotate()` helper draws each tight box on the photo, numbered, orange for review and blue for pass.

## Results / what the cookbook reports

`jev-1.14.0`, 2026-10-15 (raw/docs/cookbooks__ocr_confidence_bounding_boxes.md):

| Photo | Field | Extracted | Page check | Crop check | Verdict |
|---|---|---|---|---|---|
| milkwood | supplier_name | `MILKWOOD mushrooms` | 0.98 | 0.95 | ok |
| milkwood | invoice_number | `AN400074` (printed `IN400074`) | 0.88 | **0.18** | REVIEW |
| milkwood | invoice_date | `01/09/2026` | 0.53 | 0.82 | ok |
| milkwood | amount_due | `504.70` | 0.95 | 0.80 | ok |
| milkwood | bank_name | `Standard Bank` | 0.98 | 0.53 | ok |
| milkwood | account_holder | `MILKWOOD` | 0.90 | 0.57 | ok |
| milkwood | account_number | `080 11 4941` | 0.35 | 0.92 | ok |
| siyakhula | supplier_name | `Siyakhula Trading Services (Pty) Ltd ta Siyakhula Trading` | 0.96 | 0.31 | ok |
| siyakhula | invoice_number | `INV03070` (printed `INV03079`) | **0.02** | **0.04** | REVIEW |
| siyakhula | invoice_date | `16/09/2028` (printed `16/09/2026`) | **0.17** | 0.66 | REVIEW |
| siyakhula | amount_due | `R1,305.65` | 0.97 | 0.84 | ok |
| siyakhula | bank_name | `Standard Bank` | 0.97 | 0.98 | ok |
| siyakhula | account_holder | `Siyakhula Trading` | 0.93 | 0.96 | ok |
| siyakhula | account_number | `10 13 205 260 5` | 0.96 | 0.78 | ok |
| siyakhula | branch_code | `051001` | 0.97 | 0.94 | ok |

The three misreads were all flagged, and the 12 correct values all passed. "Neither check catches all three misreads alone": the crop check caught `AN400074` (0.18), which the page check passed at 0.88; the page check caught `16/09/2028` (0.17), which the crop check passed at 0.66; both caught `INV03070`. Upstream explains the page check's miss: a whole-page photo "shows small print in little detail, so these first boxes are rough". The closest correct values to the bar were the Milkwood account number (0.35, page check) and the Siyakhula supplier name (0.31, crop check). Two invoices and 15 fields are a demo, not an accuracy estimate (inferred).

## Adapting it to a new domain

- **Swap the extractor and the fields.** Any extractor works; write one `FIELDS` description per field in the words a reader would use, and keep `SAME_VALUE` (or your own equivalence rule) so formatting differences do not fail a correct value.
- **Tune `BAR` on your own documents.** 0.25 is this cookbook's choice. TypeSafe's general Noul starting point is 0.7 for yes and 0.3 for no, with values between going to a person ([[concepts/noul]]); here a field needs to fail only one of two checks, so the bar sits lower (inferred). Set it from the cost of a missed misread vs. a wasted review, on more than two invoices.
- **Keep both checks.** The results show they fail on different fields; drop one and you lose a catch.
- **Tune the zoom and pad.** `ZOOM`, `ZOOM_MIN` and `GROW` fit invoice photos; small print on large pages may need a bigger minimum region or more upscaling. The Box page's own advice is the same two moves: ask again on a higher-resolution crop, and pad before cropping ([[concepts/box]]).
- **Other uses (inferred):** receipts, delivery notes, ID or form fields, meter readings: anything where an extractor's value can be checked against where it is printed.

## Gotchas

- **Experimental feature.** Request format, answers and limits can change between releases; the cookbook's numbers are from one model on one date ([[concepts/images]]).
- **Header on every request.** Without `TypeSafe-Danger-Zone-Images: true` an image or Box request returns `400`; the SDK does not add it, so the cookbook sets it on the client.
- **Box always returns a box.** There is no "not found" answer ([[concepts/box]]); here the Noul checks, not the box, decide review.
- **`attach_assets` is required for images inside text.** Without the `with` block the reference goes as plain text ([[concepts/images]]).
- **Size limits are yours to meet.** The API rejects images outside 280–1568 px per side and below 78,400 total pixels; resizing, upscaling and the white canvas are client code, and box coordinates must be mapped back through that canvas.
- **Each image in `instructions` is sent with its question.** Fine here (one crop per question); an image several questions share belongs in `state` once ([[concepts/images]]).
- **Not an OCR engine.** Jev checks a given value; it never returns the corrected text (`IN400074` comes from the docs' table, not from Jev).

## Related

- [[concepts/images]] — the `assets` map, header, limits and `attach_assets`
- [[concepts/box]] — Box questions, coordinates and confidence
- [[concepts/noul]] — the yes/no checks and starting thresholds
- [[cookbooks/reading-scanned-documents]] — Noul + Box + Choice on a scanned record
- [[cookbooks/feeder-watch]] — Box and Nouls on video frames
- [[cookbooks/citation-check]] — checking a claimed value against its source text
- [[patterns/confidence-routing]] — sending low scores to a person
- [[cookbooks/overview]] — the cookbook catalog

## Sources

- raw/docs/cookbooks__ocr_confidence_bounding_boxes.md (https://docs.typesafe.ai/cookbooks/ocr_confidence_bounding_boxes); re-fetched 2026-11-03: setup no longer says to download the photos, `extractions.json` and `json_cache.json` by hand (the code fetches the first three with `download()`; no published cache is mentioned); requests 2 and 3 now go through a shared `ask_with_images()` with an `{image}` placeholder; `region_around` moved into `check_invoice`; `GROW` is passed in. Same three requests, questions, `BAR`, model, date and every result unchanged.
- raw/docs/cookbooks.md (https://docs.typesafe.ai/cookbooks) — section and level
- raw/docs/images.md (https://docs.typesafe.ai/images) — header, limits, no server-side resize
- raw/docs/primitives__box.md (https://docs.typesafe.ai/primitives/box) — no "not found" answer, crop and pad advice
- raw/docs/primitives__noul.md (https://docs.typesafe.ai/primitives/noul) — 0.7 / 0.3 starting thresholds
- raw/docs/sdk__python__usage__basic.md (https://docs.typesafe.ai/sdk/python/usage/basic) — `attach_assets`