--- title: "Cookbook: Reading a scanned vaccination record" type: cookbook tags: [cookbook, images, box, extraction, documents, noul] created: 2026-10-29 updated: 2026-11-03 confidence: high sources: - raw/docs/cookbooks__reading_scanned_documents.md - raw/docs/cookbooks.md - raw/docs/images.md jev_version: "jev-1.14.0" sdk_python: "0.7.5" # the cookbook installs typesafe-sdk>=0.7.5,<0.8.0 and cooksafe>=0.2.0,<0.3.0 and pins model jev-1.14.0 summary: "Read a scanned vaccination record: Noul + Box per known vaccine, a year Choice with an unreadable option, then a text-only Noul per dose for 'still covers the trip'; code flags rows for a nurse." --- # Cookbook: Reading a scanned vaccination record > **TL;DR** Jev does not extract a list from a scan; it answers questions about it. Request 1 asks, for each of the 16 vaccines the clinic gives, a `Noul` (is it in the table, with exactly this name?) and a `Box` (where is its row?), plus one `Noul` for "any vaccine not on this list?". Request 2 asks, per found row, a year `Choice` whose options end with `other` and `unreadable`, and a legibility `Noul`. Request 3 sends the readings as **text, no image**, with the year-difference worked out in code, and asks a `Noul` per dose: will it still protect on the departure date? Code flags rows for a nurse. On the sample record (`jev-1.14.0`) Jev answered `unreadable` for the rabies date under a sticker and scored the five-year-old typhoid dose 0.38; both rows were flagged, nothing else was. Images and Box are experimental and need the `TypeSafe-Danger-Zone-Images: true` header. ## Goal Scanned vaccination records with no text layer need checking before a trip: which vaccines are recorded, when each dose was given, and whether each dose still covers the travel date. The cookbook reads one travel vaccination record into one row per vaccine and flags the rows a nurse should check (raw/docs/cookbooks__reading_scanned_documents.md). The sample record has two such rows: the rabies date is under a batch sticker, and the typhoid dose is from June 2021, which the cookbook says protects "for about three years", so it will not cover a trip in November 2026. Upstream lists it under "Image processing", level Intermediate, next to [[cookbooks/feeder-watch]] and [[cookbooks/ocr-confidence-bounding-boxes]] (raw/docs/cookbooks.md; [[cookbooks/overview]]). **When to use it:** documents where the possible answers are known in advance (the items a clinic, insurer or school can record; a range of plausible years) and an uncertain reading should go to a person. **When not to:** open-ended transcription of whatever is on the page, or a list of candidate items so long the questions will not fit one request (the cookbook flags this for "a list of thousands"; see [[reference/models-and-pricing]] for context length). Scanning in, and filing the result in the traveller's record, are out of scope. ## Inputs / state shape Requests 1 and 2 carry the scan as `{"page": Image(PAGE)}`; every question names `` `page` ``, and all questions in a request see the same image, so each step's questions go in one request. Request 3 carries no image (and so needs no images header): a short context string and one entry per readable row. The scan (`vaccination_record.jpg`) is in the typesafe-ai/typesafe-public-examples repo; since the 2026-11-03 docs the script fetches it itself (`download()`, below) unless it is already on disk. Responses are cached in a local `json_cache.json`, so a re-run replays them; until the 2026-11-03 docs the cookbook also published that cache in the same repo, to replay the published answers without calling the API. Accepted images are PNG or JPEG up to 16 MiB, 280 to 1568 pixels per side, so scale a high-resolution scan down first ([[concepts/images]]). The cookbook pins `TYPESAFE_MODEL = "jev-1.14.0"` and passes it on every call; it does not give a date for its run. `jev-latest` has pointed to `jev-1.14.0` since 2026-10-24 ([[reference/models-and-pricing]]). Setup, verbatim except the plotting imports: ```python import hashlib import json import os import urllib.request from datetime import date from pathlib import Path from time import perf_counter from cooksafe import JsonCache from typesafe_sdk import Image, TypeSafeClient TYPESAFE_MODEL = "jev-1.14.0" IMAGES_ON = {"TypeSafe-Danger-Zone-Images": "true"} # opt in to the image preview READ_CUTOFF = 0.90 # below this, a person checks the reading against the paper LEGIBLE_CUTOFF = 0.50 # the legibility Noul gets its own, lower cutoff IN_DATE_CUTOFF = 0.50 # below this, a nurse checks whether the dose still covers the trip client = TypeSafeClient( api_key=os.environ.get("TYPESAFE_API_KEY", "cache-only"), # keyless kernels replay base_url=os.environ.get("TYPESAFE_ENDPOINT"), timeout=120.0, ) json_cache = JsonCache(Path("json_cache.json")) EXAMPLES = ( "https://raw.githubusercontent.com/typesafe-ai/typesafe-public-examples/main/" "cookbooks/reading_scanned_documents" ) def download(name: str) -> Path: """Fetch one of this cookbook's files from typesafe-public-examples unless it is on disk.""" path = Path(name) if not path.exists(): path.parent.mkdir(parents=True, exist_ok=True) urllib.request.urlretrieve(f"{EXAMPLES}/{name}", path) return path PAGE = download("vaccination_record.jpg") PAGE_SHA = hashlib.sha256(PAGE.read_bytes()).hexdigest()[:16] # cache key for the scan @json_cache def ask(page_sha: str, questions_json: str) -> dict: """One request: the scan as the state, plus a set of questions about it.""" started = perf_counter() response = client.system_one( state={"page": Image(PAGE)}, questions=json.loads(questions_json), model=TYPESAFE_MODEL, extra_headers=IMAGES_ON, ) return { "answers": {k: a.model_dump(mode="json") for k, a in response.answers.items()}, "seconds": round(perf_counter() - started, 2), "input_tokens": response.usage.input_tokens or 0, "output_tokens": response.usage.output_tokens or 0, } def ask_page(questions: dict) -> dict: if not questions: # nothing to ask, such as a page with no rows found return {"answers": {}} return ask(PAGE_SHA, json.dumps(questions, sort_keys=True)) ``` Install upstream: `pip install ipython matplotlib 'typesafe-sdk>=0.7.5,<0.8.0' 'cooksafe>=0.2.0,<0.3.0'`, then set `TYPESAFE_API_KEY`. `TYPESAFE_ENDPOINT` is a cookbook convention, not an SDK variable ([[reference/environment-variables]]). Questions are plain dicts here, not SDK classes. ## Questions asked **Request 1: find the rows (33 questions).** The list deliberately includes look-alikes (the combined hepatitis A and B vaccine next to Hepatitis A) to check that Jev matches the exact name. The `unlisted` question uses structured `instructions` (a dict with `question` and `list`; [[concepts/advanced-structure]]) to catch a vaccine no other question would. ```python VACCINES = [ "Hepatitis A", "Hepatitis B", "Hepatitis A and B combined", "Typhoid", "Hepatitis A and typhoid combined", "Yellow fever", "Rabies", "Japanese encephalitis", "Tick-borne encephalitis", "Cholera", "MenACWY", "MenB", "Tetanus, diphtheria and polio (Td/IPV)", "MMR", "Varicella", "Influenza", ] PRESENT_CUTOFF = 0.5 # P(recorded) at or above this counts the vaccine as on the page def find_questions() -> dict: questions = {"unlisted": { "type": "noul", "instructions": { "question": "Does the vaccines table in `page` record any vaccine that is not " "in this list?", "list": VACCINES, }, }} for i, vaccine in enumerate(VACCINES): questions[f"vac_{i:02d}"] = { "type": "noul", "instructions": f"Is {vaccine} recorded in the vaccines table in `page`, with " "exactly this name?", } questions[f"vac_{i:02d}|box"] = { "type": "box", "instructions": f"the whole {vaccine} row of the vaccines table in `page`", } return questions ``` **Request 2: read the year (two questions per found row).** The in-date check only needs roughly how old a dose is, so the `Choice` asks for the year alone. A Choice always picks an option, so without `unreadable` the date under the sticker "would otherwise get whichever year looked closest". The legibility `Noul` is a second, separate signal that something covers the date. The record writes day/month/year, so the instruction says the year is the last part. ```python YEAR = { **{f"y{n}": str(n) for n in range(2016, 2027)}, "other": "A year before 2016 or after 2026", "unreadable": "The year is covered, stained, smudged or damaged and can't be read", } def read_questions(row: dict) -> dict: where = f"the DATE GIVEN cell of the {row['vaccine']} row of the vaccines table in `page`" i = row["id"] return { f"{i}|year": { "type": "choice", "criteria": YEAR, "instructions": "Dates on this record are written day/month/year, so the year " f"is the last part. What year is written in {where}?", }, f"{i}|legible": { "type": "noul", "instructions": f"Is the date in {where} fully legible, with nothing covering, " "staining, smudging or blurring any of it?", }, } ``` **Request 3: does each dose cover the trip (text only, one `Noul` per readable row).** The reading stays with the image questions; this request judges only the dose. Date arithmetic is "a job for code", so code turns each year into a rough "given before departure" phrase, and rows without a year are left out. ```python DEPARTURE = date(2026, 11, 12) # from the traveller's booking def before_departure(year: int) -> str: years = DEPARTURE.year - year return "less than a year" if years == 0 else f"about {years} year{'s' * (years != 1)}" CHECK_CONTEXT = ( "Each entry is one line of an adult traveller's vaccination record, as read from a " f"scan. The traveller leaves on {DEPARTURE.day} November {DEPARTURE.year}. Judge " "only whether the dose will still protect the traveller on that date, by usual " "travel-health guidance for adults." ) IN_DATE_QUESTION = "Will this dose still protect the traveller on the departure date?" IN_DATE_CRITERIA = { "true": "The dose is within the usual protection period for this vaccine on the " "departure date.", "false": "The dose is past the usual protection period for this vaccine by the " "departure date.", } def check_questions(readable: list[dict]) -> dict: return { row["id"]: { "type": "noul", "instructions": {"question": IN_DATE_QUESTION, "entry": row["id"]}, "criteria": IN_DATE_CRITERIA, } for row in readable } def check_state(readable: list[dict]) -> dict: return { "context": CHECK_CONTEXT, "entries": [{"id": r["id"], "vaccine": r["vaccine"], "year_given": as_text(r), "given_before_departure": before_departure(year_given(r))} for r in readable], } ``` (`as_text` and `year_given` turn the Choice key back into a year: `None` for `other` or `unreadable`, else `int(key[1:])` on keys such as `y2021`.) Request 3 goes through a separate cached `check(state_json, questions_json)` that calls `client.system_one(state=..., questions=..., model=TYPESAFE_MODEL)` with no header. Until the 2026-11-03 docs `ask()` and `check()` also took the model as their first argument (so it was part of the cache key); they now read `TYPESAFE_MODEL`. ## Combining logic in code - **Present** if the vaccine's `Noul` is at or above `PRESENT_CUTOFF` (0.5). Only then is its box used ("A `Box` always returns a box, even for a vaccine that isn't on the page"). Found rows are sorted by `box["top"]`, which gives page order, and numbered `row-1`, `row-2`, ... - **Check the reading** if the row's reading score, `min(present, year_confidence)` (the presence Noul and the year Choice's `confidence`), is below `READ_CUTOFF` (0.90), or the legibility Noul is below `LEGIBLE_CUTOFF` (0.50), or the year came back `other` or `unreadable`. - **Check the dose** if the in-date Noul is below `IN_DATE_CUTOFF` (0.50). - **Check the page** if `unlisted` is at or above 0.5, since that vaccine has no row of its own. A high in-date score does not cancel a doubtful reading, because it was judged on a year that might be wrong. The cutoffs are "a starting point"; tune them on records a person has already checked. ```python def reading_score(row: dict) -> float: """The weakest of the answers that say the row was found and read.""" return min(row["present"], row["year_confidence"]) def review_reasons(row: dict) -> list[str]: reasons = [] if (unresolved(row) or reading_score(row) < READ_CUTOFF or row["legible"] < LEGIBLE_CUTOFF): reasons.append("Check the reading") if row["in_date"] is not None and row["in_date"] < IN_DATE_CUTOFF: reasons.append("Check the dose") return reasons ``` For each flagged row the nurse gets a crop of the page from request 1's box: the edges are fractions of the image's height and width, so multiply by the pixel size; pad by 0.02 of the page, or 0.05 when the box `confidence` is below 0.6. ## Results / what the cookbook reports All from `jev-1.14.0` (raw/docs/cookbooks__reading_scanned_documents.md). **Request 1:** the six vaccines on the page scored 0.97 or more; the ten not on it, look-alikes included, 0.02 or less; `unlisted` 0.02. Sorted by box top, the six rows came out in page order. The boxes of absent vaccines still came back, all with a top between 0.33 and 0.36. | Vaccine | Recorded | Year read | Legible | In date | Flag | |---|---|---|---|---|---| | Hepatitis A | 0.98 | 2026 | 0.96 | 0.94 | – | | Hepatitis B | 0.98 | 2026 | 0.86 (highlighter stroke across the row) | 0.83 | – | | Typhoid | 0.98 | 2021 | 0.96 | 0.38 | Check the dose | | Yellow fever | 0.98 | 2025 | 0.96 | 0.95 | – | | Tetanus, diphtheria and polio (Td/IPV) | 0.98 | 2024 | 0.96 | 0.93 | – | | Rabies | 0.97 | `unreadable` | 0.05 | not asked | Check the reading | Five years were read correctly. The legibility Noul scored clear dates 0.86 to 0.96, lower than the Choice answers, which is why it gets its own 0.50 cutoff. The lowest reading score in the flag table is 0.97 (Rabies), so `READ_CUTOFF` did not trigger anything; the Rabies flag came from `unreadable` and legibility 0.05. Crops: Typhoid box top 0.48, bottom 0.53, confidence 0.88; Rabies top 0.69, bottom 0.75, confidence 0.77; each crop shows the whole row. The page was not flagged. **Appendix: why reading and checking are separate requests.** Asking the in-date question straight off the image (departure date read from the page) gave Hepatitis A 0.88, Hepatitis B 0.85, Typhoid 0.50, Yellow fever 0.96, Td/IPV 0.84 and Rabies 0.88. Both bad rows passed: Rabies at 0.88 with its date under a sticker, and Typhoid at 0.50, "right on the cutoff" (the check is `< 0.50`, so 0.50 passes). With one answer covering two steps "you can't tell whether Jev failed to read the date or judged the dose wrongly". ## Adapting it to a new domain - **Swap the lists:** your own items for `VACCINES` and value options for `YEAR`; keep `other` and `unreadable` in every list of options, and rewrite the in-date question for your documents (upstream). - **Keep look-alikes in the item list** and the "exactly this name" wording, so near-names do not match. - **Keep one "anything not on this list?" Noul** for whole-page review. - **Long item lists:** narrow first, for example with one `Noul` per group of items, then ask per item within the groups that come back high (upstream's suggestion). - **Do the arithmetic in code** and send Jev the result in words ("about 5 years") rather than two dates ([[concepts/jaggedness-jev-1-13]] lists date arithmetic as a weak spot). - **Other documents** (inferred): insurance cards, certificates, inspection forms, any scanned table whose rows come from a known list. ## Gotchas - **Experimental feature.** Image requests need `TypeSafe-Danger-Zone-Images: true` ("the request format can change between releases"); the SDK does not add it ([[reference/python-sdk-questions]]). Box needs Jev 1.14 ([[concepts/box]]). - **A Box always answers;** gate it on the Noul. Box `confidence` may also be `null` per the schema ([[concepts/box]]); the crop code compares it with 0.6 directly (inferred risk). - **A Choice always picks.** Without `unreadable` an obscured value gets a confident-looking year. - **Choice `confidence` is not a probability** ([[concepts/confidence]]); the reading score mixes it with a Noul probability under one 0.90 cut, so tune that cut on your data. - **The in-date Noul relies on Jev's general knowledge** of protection periods ("usual travel-health guidance for adults"); the cookbook routes low scores to a nurse rather than acting on them. Thresholds here are 0.5; TypeSafe's general Noul starting point is 0.7 for yes and 0.3 for no ([[concepts/noul]]). - **No run date is given** for the numbers; they are tied to `jev-1.14.0` only. ## Related - [[concepts/images]] — image input, limits and the experimental header - [[concepts/box]] — Box answers, edges and crops - [[concepts/noul]] — presence, legibility and in-date questions - [[concepts/choice]] — the year Choice and its escape options - [[cookbooks/feeder-watch]] — the sibling image cookbook: Noul + Box per frame - [[cookbooks/ocr-confidence-bounding-boxes]] — checking OCR values with Box crops - [[cookbooks/date-extraction]] — dates from text, not scans - [[cookbooks/overview]] — the cookbook catalog ## Sources - raw/docs/cookbooks__reading_scanned_documents.md (https://docs.typesafe.ai/cookbooks/reading_scanned_documents); re-fetched 2026-11-03 (published-cache paragraph dropped, scan fetched by `download()`, `ask`/`check` no longer take the model; questions, cutoffs and results unchanged) - raw/docs/cookbooks.md (https://docs.typesafe.ai/cookbooks) — section and level - raw/docs/images.md (https://docs.typesafe.ai/images) — image limits and header