$jevwiki.ai#an LLM wiki about Jev, written for agents rather than people

Agents: read the raw Markdown of this page, or start at llms.txt.

~/wiki/cookbooks

Cookbook: OCR confidence and bounding boxes

[ cookbook ][ updated 2026-11-03 ][ confidence high ][ jev-1.14.0 ][ python sdk 0.7.5 ]#cookbook · images · box · noul · documents

TL;DR Take the values an OCR-plus-LLM extractor read from invoice photos and check each one with Jev in three requests per invoice: (1) the whole photo with a Box (where is the field?) and a Noul (is this value that field on the page? = the page check); (2) a Box per field on an enlarged region around the first box, for a tighter box; (3) a Noul per field on a crop of that box (does the crop show the value? = the crop check). A field scoring below 0.25 on either check goes to review. On two supplier invoices with three misread values, all three were flagged and no correct value was; neither check alone caught all three. Images and Box are experimental and need the TypeSafe-Danger-Zone-Images: true header and Jev 1.14.

Goal

OCR on invoice photos is "mostly right", but checking every value by hand "takes as long as typing them in" (raw/docs/cookbooks__ocr_confidence_bounding_boxes.md). The cookbook makes Jev the checker: it boxes each extracted value on the photo, scores it twice, and sends only low scorers to a person, with the box drawn so the reviewer sees each value where it is printed.

Upstream lists it under the new "Image processing" section, level Intermediate (raw/docs/cookbooks.md; Cookbooks overview). With Cookbook: Reading a scanned vaccination record and Cookbook: Feeder-cam visit log, it is one of the first three official worked examples of images and Box (all added 2026-10-29).

Experimental. The docs call images and Box "dreamer-ready": "not yet ready for production", and "their request format and answers can change between releases". They need Jev 1.14, and every request needs the TypeSafe-Danger-Zone-Images: true header; the cookbook's client sets it (Images and assets (experimental)).

When to use it: you already extract fields from scans or photos (Tesseract, Apple Vision, an LLM) and want a second opinion that points a person at the likely misreads. When not to: as the extractor itself (the cookbook starts from the extracted values; Jev never reads the text out), or as an unreviewed production gate while images are experimental.

Inputs / state shape

The state differs by request: request 1 sends {"page": <the photo>}; requests 2 and 3 send only a one-line text task, and every image goes inside its own question's instructions.

Numbers below came from jev-1.14.0 on 2026-10-15 (the cookbook pins TYPESAFE_MODEL = "jev-1.14.0"; since the 2026-10-24 alias move jev-latest is also jev-1.14.0, Models, aliases, pricing, rate limits, context). Every response is cached in a local json_cache.json, so a re-run reuses it. Until the 2026-11-03 docs the cookbook also published that cache in typesafe-ai/typesafe-public-examples, so a run reproduced the published numbers without calling the API, and said to delete it to send every request again; both lines are gone.

import io
import json
import os
import urllib.request
from pathlib import Path

from cooksafe import JsonCache
from IPython.display import display
from PIL import Image as PILImage
from PIL import ImageDraw, ImageFont
from typesafe_sdk import Box, Image, Noul, SystemOneResponse, TypeSafeClient

TYPESAFE_MODEL = "jev-1.14.0"  # Box questions need Jev 1.14
BAR = 0.25  # a check scoring below this flags the field
ZOOM = 3  # the zoomed region is three times the first box, and at least ZOOM_MIN of the page
ZOOM_MIN = (0.30, 0.15)  # width, height as fractions of the page
GROW = (0.25, 0.20)  # review margin around each box, as fractions of its width and height
MIN_SIDE, MAX_SIDE = 280, 1568  # the pixel range Jev accepts for each side of an image

client = TypeSafeClient(
    api_key=os.environ.get("TYPESAFE_API_KEY", "cache-only"),  # keyless kernels replay the cache
    base_url=os.environ.get("TYPESAFE_ENDPOINT"),
    headers={"TypeSafe-Danger-Zone-Images": "true"},
    timeout=120.0,
)
json_cache = JsonCache(Path("json_cache.json"))


EXAMPLES = (
    "https://raw.githubusercontent.com/typesafe-ai/typesafe-public-examples/main/"
    "cookbooks/ocr_confidence_bounding_boxes"
)


def download(name: str) -> Path:
    """Fetch one of this cookbook's files from typesafe-public-examples unless it is on disk."""
    path = Path(name)
    if not path.exists():
        path.parent.mkdir(parents=True, exist_ok=True)
        urllib.request.urlretrieve(f"{EXAMPLES}/{name}", path)
    return path


FIELDS = {
    # field: what it is, in the words every question uses
    "supplier_name": "the supplier's (seller's) business name",
    "invoice_number": "the invoice or document number",
    "invoice_date": "the invoice date",
    "amount_due": "the final total amount payable",
    "bank_name": "the name of the supplier's bank in the payment details",
    "account_holder": "the account holder name in the supplier's bank details",
    "account_number": "the supplier's bank account number",
    "branch_code": "the supplier's bank branch code",
}
SAME_VALUE = "Ignore differences in spacing, punctuation, currency symbols and letter case."
INPUTS = Path("inputs")  # the photos, keyed in extractions.json by file name
EXTRACTIONS = json.loads(download("extractions.json").read_text())
for photo in EXTRACTIONS:
    download(f"inputs/{photo}")

(Upstream verbatim, reordered into one block. Install: pip install ipython pillow 'typesafe-sdk[pil]>=0.7.5' 'cooksafe>=0.2.0,<0.3.0', then set TYPESAFE_API_KEY. Since the 2026-11-03 docs the code fetches extractions.json and the two photos (into inputs/) from typesafe-public-examples itself unless they are on disk; until then you downloaded them, with json_cache.json, by hand.)

Image sizing in code. Jev accepts 280 to 1568 pixels per side (Images and assets (experimental)). The cookbook's as_image() shrinks larger pictures with thumbnail, enlarges small crops (upscale_to) "so their text is readable", and pastes anything still under 280 px into the middle of a white canvas, returning the picture's frame on that canvas so a box drawn on the canvas can be mapped back to the crop. A crop(photo, box, pad) helper cuts a fractional box out of the photo. The API itself never resizes: an out-of-range image is rejected (raw/docs/images.md).

def as_image(picture: PILImage.Image, upscale_to: int | None = None) -> tuple[Image, tuple]:
    picture = picture.convert("RGB")
    if upscale_to and max(picture.size) < upscale_to:
        scale = upscale_to / max(picture.size)
        picture = picture.resize(
            (round(picture.width * scale), round(picture.height * scale)), PILImage.LANCZOS
        )
    picture.thumbnail((MAX_SIDE, MAX_SIDE))
    frame = (0.0, 0.0, 1.0, 1.0)  # left, top, width, height of the picture on the canvas
    if min(picture.size) < MIN_SIDE:
        canvas = PILImage.new("RGB", (max(picture.width, MIN_SIDE), max(picture.height, MIN_SIDE)), "white")
        x, y = (canvas.width - picture.width) // 2, (canvas.height - picture.height) // 2
        canvas.paste(picture, (x, y))
        frame = (x / canvas.width, y / canvas.height, picture.width / canvas.width,
                 picture.height / canvas.height)
        picture = canvas
    buffer = io.BytesIO()
    picture.save(buffer, "JPEG", quality=90)
    return Image(buffer.getvalue()), frame


def crop(photo: str, box: dict, pad: float = 0.0) -> PILImage.Image:
    with PILImage.open(INPUTS / photo) as page:
        w, h = page.size
        return page.convert("RGB").crop((
            max(0, int((box["left"] - pad) * w)), max(0, int((box["top"] - pad) * h)),
            min(w, int((box["right"] + pad) * w)), min(h, int((box["bottom"] + pad) * h)),
        ))

(Upstream verbatim.)

Questions asked

Request State Per field type instructions (verbatim template)
1 · the photo {"page": <photo>} box_<field> box {FIELDS[field]}, printed as "{value}" in `page`
1 same fits_<field> noul Is "{value}" {FIELDS[field]} on `page`? {SAME_VALUE}
2 · zoomed regions {"task": "Locate fields in zoomed-in parts of a supplier invoice."} <field> box {FIELDS[field]}, printed as "{value}" in this image: {image}
3 · crops {"task": "Check values read from cropped parts of a supplier invoice."} <field> noul Image: {image}\nDoes this cropped image clearly show {FIELDS[field]} as "{value}"? {SAME_VALUE} Answer no if it is cut off, unreadable or different.

(Upstream uses typographic quotes, “ ”, around {value}.) No question has criteria: Box takes none, and the Nouls use plain yes/no. The extracted value is in every Box question as a hint "about which text to box". The page check is request 1's Noul P(yes); the crop check is request 3's.

Request 1 (verbatim):

def page_questions(values: dict) -> dict:
    questions = {}
    for field, value in sorted(values.items()):
        questions[f"box_{field}"] = Box(
            instructions=f"{FIELDS[field]}, printed as “{value}” in `page`"
        )
        questions[f"fits_{field}"] = Noul(
            instructions=f"Is “{value}” {FIELDS[field]} on `page`? {SAME_VALUE}"
        )
    return questions


def as_json(questions: dict) -> str:
    return json.dumps(
        {key: q.model_dump(mode="json", exclude_none=True) for key, q in questions.items()}
    )


@json_cache
def find_fields(photo: str, questions_json: str) -> dict:
    questions = json.loads(questions_json)
    fields = sorted(key.removeprefix("box_") for key in questions if key.startswith("box_"))
    with PILImage.open(INPUTS / photo) as page:
        image, _ = as_image(page)  # a whole page fills the size range, so no canvas
    response = client.system_one(
        state={"page": image}, questions=questions, model=TYPESAFE_MODEL
    )
    return {
        "boxes": {f: response.boxes[f"box_{f}"].box.model_dump() for f in fields},
        "fits": {f: response.nouls[f"fits_{f}"].noul for f in fields},
    }

Request 2 cuts a region around each first box (three times its size, at least 30% of the page wide and 15% high, "so the field stays inside it even if the first box was off"), enlarges it to 1200 px, and puts each region's image in its own question; client.attach_assets(...) sends them all in one request and resolves the image references inside the text (Python SDK: install, clients, system_one()). Since the 2026-11-03 docs the questions are built with a literal {image} placeholder (IMAGE), and ask_with_images() swaps in each field's image reference (str(images[field])) just before sending, so the cached functions take the questions as JSON without image data; until then each question embedded its image when it was built, and the cached functions took the extracted values and boxes instead. The code then maps each new box from canvas to region to page and pads it by GROW (25% of its width, 20% of its height on each side) "so that cropping to it can't cut off part of the value" (verbatim):

IMAGE = "{image}"  # stands for each question's own image until the request is sent
ZOOM_TASK = "Locate fields in zoomed-in parts of a supplier invoice."


def region_around(box: dict) -> dict:
    width = min(1.0, max(ZOOM_MIN[0], (box["right"] - box["left"]) * ZOOM))
    height = min(1.0, max(ZOOM_MIN[1], (box["bottom"] - box["top"]) * ZOOM))
    left = min(max(0.0, (box["left"] + box["right"] - width) / 2), 1 - width)
    top = min(max(0.0, (box["top"] + box["bottom"] - height) / 2), 1 - height)
    return {"left": left, "top": top, "right": left + width, "bottom": top + height}


def zoom_questions(values: dict) -> dict:
    return {
        field: Box(
            instructions=f"{FIELDS[field]}, printed as “{values[field]}” in this image: {IMAGE}"
        )
        for field in sorted(values)
    }


def ask_with_images(task: str, questions: dict, images: dict) -> SystemOneResponse:
    """Put each field's image into its question, and send all the questions in one request."""
    for field, question in questions.items():
        question["instructions"] = question["instructions"].replace(IMAGE, str(images[field]))
    with client.attach_assets(list(images.values())):  # resolves the references in the text
        return client.system_one(state={"task": task}, questions=questions, model=TYPESAFE_MODEL)


@json_cache
def zoom_fields(
    photo: str, regions_json: str, task: str, questions_json: str, grow: tuple
) -> dict:
    regions, images, frames = json.loads(regions_json), {}, {}
    for field, region in regions.items():
        images[field], frames[field] = as_image(crop(photo, region), upscale_to=1200)
    response = ask_with_images(task, json.loads(questions_json), images)
    tight = {}
    for field, region in regions.items():
        b, (fx, fy, fw, fh) = response.boxes[field].box, frames[field]
        # canvas -> region -> page, then add the review margin
        left, right = (b.left - fx) / fw, (b.right - fx) / fw
        top, bottom = (b.top - fy) / fh, (b.bottom - fy) / fh
        gx, gy = (right - left) * grow[0], (bottom - top) * grow[1]
        rw, rh = region["right"] - region["left"], region["bottom"] - region["top"]
        tight[field] = {
            "left": max(0.0, region["left"] + (left - gx) * rw),
            "top": max(0.0, region["top"] + (top - gy) * rh),
            "right": min(1.0, region["left"] + (right + gx) * rw),
            "bottom": min(1.0, region["top"] + (bottom + gy) * rh),
        }
    return {"boxes": tight}

Request 3 (verbatim):

CROP_TASK = "Check values read from cropped parts of a supplier invoice."


def crop_questions(values: dict) -> dict:
    return {
        field: Noul(
            instructions=f"Image: {IMAGE}\nDoes this cropped image clearly show {FIELDS[field]}"
            f" as “{values[field]}”? {SAME_VALUE} Answer no if it is cut off, unreadable or"
            " different."
        )
        for field in sorted(values)
    }


@json_cache
def check_crops(photo: str, boxes_json: str, task: str, questions_json: str) -> dict:
    images = {
        field: as_image(crop(photo, box, pad=0.01), upscale_to=800)[0]
        for field, box in json.loads(boxes_json).items()
    }
    response = ask_with_images(task, json.loads(questions_json), images)
    return {
        "shows": {f: response.nouls[f].noul for f in sorted(images)},
    }

Combining logic in code

Jev returns boxes and probabilities; the review rule is one line of code. A field goes to review when either check is below BAR (verbatim):

def check_invoice(photo: str, values: dict) -> dict:
    first = find_fields(photo, as_json(page_questions(values)))
    regions = {field: region_around(box) for field, box in first["boxes"].items()}
    tight = zoom_fields(photo, json.dumps(regions, sort_keys=True), ZOOM_TASK,
                        as_json(zoom_questions(values)), GROW)
    shown = check_crops(photo, json.dumps(tight["boxes"], sort_keys=True), CROP_TASK,
                        as_json(crop_questions(values)))
    return {
        field: {
            "value": values[field],
            "box": tight["boxes"][field],
            "fits": first["fits"][field],
            "shows": shown["shows"][field],
            "review": first["fits"][field] < BAR or shown["shows"][field] < BAR,
        }
        for field in sorted(values)
    }

The Box confidence is not used in the rule: the first box only decides where to zoom, and the tight box only decides the crop and the outline a reviewer sees. An annotate() helper draws each tight box on the photo, numbered, orange for review and blue for pass.

Results / what the cookbook reports

jev-1.14.0, 2026-10-15 (raw/docs/cookbooks__ocr_confidence_bounding_boxes.md):

Photo Field Extracted Page check Crop check Verdict
milkwood supplier_name MILKWOOD mushrooms 0.98 0.95 ok
milkwood invoice_number AN400074 (printed IN400074) 0.88 0.18 REVIEW
milkwood invoice_date 01/09/2026 0.53 0.82 ok
milkwood amount_due 504.70 0.95 0.80 ok
milkwood bank_name Standard Bank 0.98 0.53 ok
milkwood account_holder MILKWOOD 0.90 0.57 ok
milkwood account_number 080 11 4941 0.35 0.92 ok
siyakhula supplier_name Siyakhula Trading Services (Pty) Ltd ta Siyakhula Trading 0.96 0.31 ok
siyakhula invoice_number INV03070 (printed INV03079) 0.02 0.04 REVIEW
siyakhula invoice_date 16/09/2028 (printed 16/09/2026) 0.17 0.66 REVIEW
siyakhula amount_due R1,305.65 0.97 0.84 ok
siyakhula bank_name Standard Bank 0.97 0.98 ok
siyakhula account_holder Siyakhula Trading 0.93 0.96 ok
siyakhula account_number 10 13 205 260 5 0.96 0.78 ok
siyakhula branch_code 051001 0.97 0.94 ok

The three misreads were all flagged, and the 12 correct values all passed. "Neither check catches all three misreads alone": the crop check caught AN400074 (0.18), which the page check passed at 0.88; the page check caught 16/09/2028 (0.17), which the crop check passed at 0.66; both caught INV03070. Upstream explains the page check's miss: a whole-page photo "shows small print in little detail, so these first boxes are rough". The closest correct values to the bar were the Milkwood account number (0.35, page check) and the Siyakhula supplier name (0.31, crop check). Two invoices and 15 fields are a demo, not an accuracy estimate (inferred).

Adapting it to a new domain

Gotchas

Sources