---
title: "Cookbook: Feeder-cam visit log"
type: cookbook
tags: [cookbook, images, box, choice, noul, video]
created: 2026-10-29
updated: 2026-11-03
confidence: high
sources:
  - raw/docs/cookbooks__feeder_watch.md
  - raw/docs/cookbooks.md
  - raw/docs/images.md
  - raw/docs/primitives__box.md
jev_version: "jev-1.14.0"
sdk_python: "0.7.5"   # the cookbook installs typesafe-sdk>=0.7.5,<0.8.0 (plus pillow, matplotlib, ipython), imports cooksafe without installing it, and pins model jev-1.14.0
summary: "Turn bird-feeder video into a visit log: one frame a second, five questions per frame (Noul, Choice, Box, two species Choices), a named Box per species when two share the feeder, visits built in code."
---

# Cookbook: Feeder-cam visit log

> **TL;DR** Send each video frame as an `Image` with five questions in one request: a `Noul` (is there a bird?), a `Choice` (none / one / several), a `Box` (the most prominent bird) and two species `Choice`s (leftmost, rightmost bird). One Box returns one box, so when two species share the feeder a second request asks for a Box per species *by name*; boxes that overlap by 0.6 or more are merged. Plain code turns the per-frame answers into visits. On a 101-second clip (`jev-1.14.0`, 2026-10-15): 178 requests, `bird_present` agreed with hand labels on 101/101 frames, the right species came back for 132 of 133 labeled birds, and both long visits matched the hand-label log to the second. Images and Box are experimental and need the `TypeSafe-Danger-Zone-Images: true` header.

## Goal

A camera on a bird feeder, a front door or a street records hours of footage. The cookbook sends Jev one frame per second, asks the same questions about each, and uses ordinary code to turn the answers into a visit log per species: who came, from when to when, for how long (raw/docs/cookbooks__feeder_watch.md).

Upstream lists it under "Image processing" ("Find things in photos and scans, and check what was read from them"), level Intermediate, next to [[cookbooks/reading-scanned-documents]] and [[cookbooks/ocr-confidence-bounding-boxes]] (raw/docs/cookbooks.md; [[cookbooks/overview]]). It is one of the first official cookbooks built on the experimental image input ([[concepts/images]]) and `Box` question ([[concepts/box]]).

**When to use it:** a fixed camera, a closed list of things you expect to see, and a per-frame yes/no plus "which one" and "where" that code can aggregate over time. **When not to:** a flock or several of the same species in one frame (a named Box cannot tell two cardinals apart), counting beyond "several", or frames below the image size floor. The cookbook's own advice for flocks is a zero-shot detector (it names Meta's SAM 3) to find every bird, then Jev to name the species of each crop.

## Inputs / state shape

Each request's `state` is one frame under the key `frame`, sent as an SDK `Image`; every question refers to `` `frame` ``:

```python
state={"frame": Image(FRAMES / frame)}
```

The footage is "Busy Bird Feeder", a 101-second public-domain clip filmed by Courtney Celley of the U.S. Fish and Wildlife Service. For the first minute a male cardinal and a downy woodpecker share the feeder; then the male cardinal leaves and a female cardinal feeds from behind the feeder, mostly out of sight. TypeSafe cut one frame a second at 1024 × 576 and hand-labeled each frame with every bird in full view: 11 frames with no bird in full view, 27 with one, 63 with two. Twenty-two frames are marked `partial` (the female cardinal cut off or mostly hidden). Frames (`usfws_frames.zip`) and labels (`usfws_labels.json`) are in the typesafe-ai/typesafe-public-examples repo. Every response is saved in a local `json_cache.json`, so a second run makes no requests; until the 2026-11-03 docs the cookbook also published that cache there, to replay the published numbers without a key. A full run sends about 180 requests.

Numbers below came from `jev-1.14.0` on 2026-10-15 (the cookbook passes `model="jev-1.14.0"` on every call, commented "Box questions need Jev 1.14"; `jev-latest` has pointed to `jev-1.14.0` since 2026-10-24, [[reference/models-and-pricing]]).

Setup, verbatim except the omitted plotting code:

```python
import json
import os
from pathlib import Path
from time import perf_counter

from cooksafe import JsonCache
from typesafe_sdk import Box, Choice, Image, Noul, SystemOneResponse, TypeSafeClient

TYPESAFE_MODEL = "jev-1.14.0"  # Box questions need Jev 1.14
IMAGE_HEADERS = {"TypeSafe-Danger-Zone-Images": "true"}  # images and Box are opt-in

client = TypeSafeClient(
    api_key=os.environ.get("TYPESAFE_API_KEY", "cache-only"),  # keyless runs replay the cache
    timeout=60.0,
)
json_cache = JsonCache(Path("json_cache.json"))  # every response, keyed by frame and questions
```

Install upstream: `pip install 'typesafe-sdk>=0.7.5,<0.8.0' pillow matplotlib ipython` (or the same with `uv add`). That line does not install `cooksafe`, which the script imports; add `'cooksafe>=0.2.0,<0.3.0'` as the sibling image cookbook does (inferred fix; [[cookbooks/reading-scanned-documents]]).

## Questions asked

Five questions, one request per frame, so the image is sent once:

| Key | `type` | `instructions` | `criteria` |
|---|---|---|---|
| `bird_present` | `noul` | "Is there at least one bird in `frame`?" | none |
| `how_many` | `choice` | "How many birds are in `frame`?" | `none` "No bird at all."; `one` "Exactly one bird."; `several` "Two or more birds." |
| `bird` | `box` | "The most prominent bird in `frame`" | none (Box takes none) |
| `species_left` | `choice` | "What species is the leftmost bird in `frame`?" | `SPECIES` (ten species plus `other`) |
| `species_right` | `choice` | "What species is the rightmost bird in `frame`?" | `SPECIES` |

With one bird in the frame both species answers describe it; with two, each describes one bird. Each species option describes what the bird looks like *on a feeder camera*, not what a field guide says: the grackle option warns it can fill the frame so only its head shows.

```python
SPECIES = {
    "blue_jay": "A blue bird with a crest, a pale grey-white face and chest, a black necklace, "
    "and black bars on the blue wings and tail.",
    "cardinal_male": "A bright red bird with a pointed crest, a black mask, and an orange beak.",
    "cardinal_female": "A warm brown or buff bird with a pointed crest, reddish tinges on the "
    "wings, tail and crest, and an orange beak.",
    "grackle": "A common grackle: a glossy black bird with a pale yellow eye, a long heavy dark "
    "beak, and an iridescent blue, purple or bronze sheen on the head and body. On a close-up "
    "feeder camera it often fills the frame, so only the glossy dark head, pale eye and beak "
    "may be visible.",
    "starling": "A European starling: a chunky short-tailed dark bird with glossy plumage "
    "speckled with pale spots, a dark eye, and a pointed yellow or dark beak. Juveniles are "
    "plain grey-brown with a dark beak.",
    "mourning_dove": "A plump grey-brown dove with a small head, a long pointed tail, and black "
    "spots on the wings.",
    "house_sparrow": "A small streaky brown sparrow. Males have a grey crown and a black bib.",
    "house_finch": "A small streaky brown finch. Males have a rosy red head and chest.",
    "red_winged_blackbird": "A glossy black bird with a red shoulder patch edged in yellow.",
    "downy_woodpecker": "A small black-and-white woodpecker with a white stripe down the back "
    "and spotted wings. Males have a red patch on the back of the head.",
    "other": "A bird that matches none of the other descriptions.",
}

QUESTIONS = {
    "bird_present": Noul(instructions="Is there at least one bird in `frame`?"),
    "how_many": Choice(
        instructions="How many birds are in `frame`?",
        criteria={
            "none": "No bird at all.",
            "one": "Exactly one bird.",
            "several": "Two or more birds.",
        },
    ),
    "bird": Box(instructions="The most prominent bird in `frame`"),
    "species_left": Choice(
        instructions="What species is the leftmost bird in `frame`?", criteria=SPECIES
    ),
    "species_right": Choice(
        instructions="What species is the rightmost bird in `frame`?", criteria=SPECIES
    ),
}
```

The call itself (verbatim): the questions are serialized to JSON so the cache can key on them, and the images header goes on every request with `extra_headers`:

```python
@json_cache
def ask_cached(frame: str, questions_json: str) -> dict:
    started = perf_counter()
    response = client.system_one(
        state={"frame": Image(FRAMES / frame)},
        questions=json.loads(questions_json),
        model=TYPESAFE_MODEL,
        extra_headers=IMAGE_HEADERS,
    )
    seconds = round(perf_counter() - started, 2)
    return {"response": response.model_dump(mode="json", exclude={"model"}), "seconds": seconds}


def ask(path: Path, questions: dict):
    as_json = json.dumps(
        {key: q.model_dump(mode="json", exclude_none=True) for key, q in questions.items()},
        sort_keys=True,
    )
    cached = ask_cached(path.name, as_json)
    # The cache leaves the model out; the response type needs one, so use the requested model.
    response = SystemOneResponse.model_validate({**cached["response"], "model": TYPESAFE_MODEL})
    return response, cached["seconds"]
```

Until the 2026-11-03 docs `ask_cached` took the model as its first argument and cached the whole response, `model` included; it now keys on frame and questions only and leaves `model` out of the cached response.

**Second request: a Box per species by name.** Asking for "the leftmost bird" and "the rightmost bird" as two Boxes failed: on frame u010 both boxes landed on the cardinal (confidence 0.76 and 0.32). Naming the species the first request found worked; where a name alone could fit another bird, the phrase adds what it looks like:

```python
BOX_TARGETS = {
    "blue_jay": "the blue jay",
    "cardinal_male": "the male northern cardinal (the bright red bird)",
    "cardinal_female": "the female northern cardinal (the brown bird with an orange beak)",
    "grackle": "the common grackle (the glossy black bird)",
    "starling": "the European starling (the speckled dark bird)",
    "mourning_dove": "the mourning dove",
    "house_sparrow": "the house sparrow",
    "house_finch": "the house finch",
    "red_winged_blackbird": "the red-winged blackbird",
    "downy_woodpecker": "the downy woodpecker (the black and white bird)",
    "other": "the bird that is none of the listed species",
}


def box_questions(species: list[str]) -> dict:
    return {sp: Box(instructions=f"{BOX_TARGETS[sp]} in `frame`") for sp in species}
```

On u010 the named boxes separated: `cardinal_male` left 0.13 right 0.42 (confidence 0.84), `downy_woodpecker` left 0.59 right 0.73 (confidence 0.36).

## Combining logic in code

Three settings decide the per-frame answer, all in code: `PRESENT_AT = 0.5`, `SPECIES_AT = 0.5`, `OVERLAP_AT = 0.6`. A Box always returns a box, even on an empty frame or for a species that is barely there, so the `Noul` decides whether a box is used, and two named boxes overlapping by 0.6 or more (intersection over union) count as one bird found twice; only the higher-confidence box is kept.

```python
OVERLAP_AT = 0.6  # two named boxes overlapping this much are one bird found twice


def overlap(a: dict, b: dict) -> float:
    """Intersection over union of two boxes given as edge fractions."""
    w = max(0.0, min(a["right"], b["right"]) - max(a["left"], b["left"]))
    h = max(0.0, min(a["bottom"], b["bottom"]) - max(a["top"], b["top"]))
    area = lambda e: (e["right"] - e["left"]) * (e["bottom"] - e["top"])  # noqa: E731
    union = area(a) + area(b) - w * h
    return w * h / union if union > 0 else 0.0


def dedupe(birds: list[dict]) -> list[dict]:
    kept: list[dict] = []
    for bird in sorted(birds, key=lambda b: -b["box_confidence"]):
        if all(overlap(bird["box"], k["box"]) < OVERLAP_AT for k in kept):
            kept.append(bird)
    return kept


PRESENT_AT = 0.5  # below this, the frame is empty and the other answers are ignored
SPECIES_AT = 0.5  # a species answer below this is too unsure to name a box


def ask_frame(path: Path) -> dict:
    first, seconds = ask(path, QUESTIONS)
    result = {
        "frame": path.name,
        "bird_present": first.nouls["bird_present"].noul,
        "how_many": first.choices["how_many"].choice,
        "birds": [],
        "requests": 1,
        "seconds": seconds,
    }
    # the species named by the leftmost and rightmost answers, each with its best probability
    named: dict[str, float] = {}
    sure_sides = 0
    for key in ("species_left", "species_right"):
        answer = first.choices[key]
        p = answer.probabilities[answer.choice]
        if p >= SPECIES_AT:
            named[answer.choice] = max(p, named.get(answer.choice, 0.0))
            sure_sides += 1
    if result["bird_present"] < PRESENT_AT or not named:
        return result

    # with several birds, the `bird` box only fits when both sides name the same species
    if result["how_many"] == "several" and (len(named) == 2 or sure_sides < 2):
        second, more_seconds = ask(path, box_questions(list(named)))
        result["requests"] = 2
        result["seconds"] += more_seconds
        boxes = {sp: second.boxes[sp] for sp in named}
    else:
        top = max(named, key=named.get)
        named, boxes = {top: named[top]}, {top: first.boxes["bird"]}

    birds = [
        {"species": sp, "p": p, "box": boxes[sp].box.model_dump(), "box_confidence": boxes[sp].confidence}
        for sp, p in named.items()
    ]
    result["birds"] = dedupe(birds)
    return result
```

The species probability used is `probabilities[choice]` (the top option's probability), not the Choice `confidence` field ([[concepts/confidence]]). Frames are independent, so the batch run uses `ThreadPoolExecutor(max_workers=4)` and `pool.map` (results stay in frame order); on a live camera, upstream says to send one frame at a time, each as soon as the previous answer is back.

**Visits** are built in frame order, one species at a time (two birds at once make two overlapping visits). A frame counts for a species at `species_at=0.6`; a visit stays open across gaps of up to `max_gap=2` frames; a visit needs `min_frames=2` frames naming its species, so a one-frame misread does not become a visit. Jev never sees these settings; re-running `visits()` on saved results costs no requests.

```python
def visits(results: list[dict], species_at=0.6, max_gap=2, min_frames=2) -> list[dict]:
    done, open_visits, last_seen = [], {}, {}
    for second, result in enumerate(results):  # one frame per second
        seen = {b["species"] for b in result["birds"] if b["p"] >= species_at}
        for sp in list(open_visits):
            if sp not in seen and second - last_seen[sp] > max_gap:
                done.append(open_visits.pop(sp))
        for sp in sorted(seen):
            if sp in open_visits:
                open_visits[sp]["end"] = second
                open_visits[sp]["frames"] += 1
            else:
                open_visits[sp] = {"species": sp, "start": second, "end": second, "frames": 1}
            last_seen[sp] = second
    done.extend(open_visits.values())
    kept = [v for v in done if v["frames"] >= min_frames]
    return sorted(kept, key=lambda v: (v["start"], v["species"]))
```

## Results / what the cookbook reports

All from `jev-1.14.0`, 2026-10-15 (raw/docs/cookbooks__feeder_watch.md).

**Three sample frames, first request:**

| Frame | Label | Seconds | `bird_present` | `how_many` | `bird` box confidence | `species_left` / `species_right` |
|---|---|---|---|---|---|---|
| u010 | cardinal_male, downy_woodpecker | 0.35 | 0.98 | `several` 1.00 | 0.71 (covers only the cardinal) | `cardinal_male` 1.00 / `downy_woodpecker` 1.00 |
| u090 | downy_woodpecker | 0.29 | 0.95 | `one` 1.00 | 0.85 | `downy_woodpecker` 0.99 / 0.99 |
| u097 | empty | 0.28 | 0.07 | `none` 0.99 | 0.14 (a box anyway) | `other` 0.38 / `other` 0.61 |

On the empty frame the species questions still named an option (`other`); the low `bird_present` is what tells code to ignore them.

**Full run:** 101 frames, 178 requests; 77 frames needed the second request, 66 ended with two boxes (the rest had only one sure species answer, or `dedupe()` merged the boxes); median 0.54 s per frame.

**Visit log:**

| Species | From the answers | From the hand labels |
|---|---|---|
| `cardinal_male` | 0:00 to 1:02 (63 s) | 0:00 to 1:02 (63 s) |
| `downy_woodpecker` | 0:00 to 1:29 (90 s) | 0:00 to 1:29 (90 s) |
| `cardinal_male` | 1:12 to 1:15 (4 s), **wrong**: the female cardinal behind the feeder, misnamed when only her red beak showed | none |

In every run TypeSafe made of the clip the two long visits came out the same; the short visits changed from run to run. With `min_frames=1`, four single-frame sightings appear: `cardinal_female` at 1:04 and 1:08 (real), `other` at 1:16, `cardinal_male` at 1:34 (a misread). A higher `min_frames` keeps out more misreads and drops more birds that show for a second or two.

**Against the labels:** `bird_present` agreed on 101/101 frames (`partial` frames count as having a bird); 132/133 labeled birds were returned (`partial` frames left out of this check). The one miss is u063 (1:02): the woodpecker, sharing the frame with the male cardinal, came back as only `cardinal_male`. The hardest presence frames were 1:33 and 1:34, where only part of a cardinal shows; Jev scored both about 0.8 and both are labeled `partial`. One failure `dedupe()` does not catch: at 1:15 Jev named a male cardinal from the female's beak, and the male-cardinal box landed on the woodpecker, overlapping the woodpecker's own box by less than 0.6, so both stayed. Upstream suggests a per-box check: crop to the box and ask a `Noul` whether the bird in the crop is that species.

**Live:** a small web app on the same questions (webcam or video file, boxes drawn, running visit log) played a stretch of the Ostdrossel feeder cam, which the questions were not tuned on, at about two answers a second (video demo; no accuracy figure given).

## Adapting it to a new domain

- **Write the option list for your camera**, describing what the camera actually shows of each target, close-ups included, and keep an `other` option so an unlisted target is not forced into a listed name.
- **More than two targets:** ask one `Noul` per target ("Is there a blue jay in `frame`?"), then a named Box for each one that comes back high (upstream's suggestion).
- **Same-kind crowds** (a flock, a queue of people): use a detector to find every instance, then Jev per crop (upstream names SAM 3 for the detector).
- **Other cameras** (inferred from the cookbook's framing): a front-door camera with "person / parcel / animal" options, a street camera with vehicle types; the presence Noul, named Boxes, overlap merge and visit rules carry over.
- **Re-tune every threshold** (`PRESENT_AT`, `SPECIES_AT`, `OVERLAP_AT`, and the `visits()` defaults) on a few minutes of your own hand-labeled footage. They "worked here". TypeSafe's general Noul starting point is 0.7 for yes and 0.3 for no, with values between going to a person ([[concepts/noul]]); this cookbook uses a single 0.5 cut.
- **Size frames** to the image limits: 280 to 1568 pixels per side and at least 78,400 pixels in total; smaller frames are rejected ([[concepts/images]]).

## Gotchas

- **Experimental feature.** Images and Box are "dreamer-ready"; every request carrying an image or a Box needs `TypeSafe-Danger-Zone-Images: true`, which the SDK does not add for you ([[reference/python-sdk-questions]]). Box runs on Jev 1.14 only; pin `jev-1.14.0` ([[concepts/box]]).
- **A Box always answers.** There is no "not found"; gate every box on a presence answer. On the empty frame u097 the box came back with confidence 0.14.
- **One Box, one box.** Position phrases ("leftmost", "rightmost") put both boxes on the same bird here; naming the target separated them.
- **The Box is paid for on every frame.** The first request always carries all five questions, including on frames code will then discard (inferred from the code).
- **Box `confidence` can be `null`** per the API schema and the SDK types ([[concepts/box]]); `dedupe()` sorts on `-b["box_confidence"]` and would fail on `None` (inferred; it did not happen in the published run).
- **Short visits are not stable.** Run to run, only the two long visits repeated; treat one- to four-second visits as candidates for review.
- **Install line omits `cooksafe`** (see Inputs).

## Related

- [[concepts/box]] — the Box question: edges, confidence, no not-found answer
- [[concepts/images]] — image input, limits and the experimental header
- [[concepts/choice]] — species and count questions; `probabilities` vs `confidence`
- [[concepts/noul]] — the presence gate and starting thresholds
- [[cookbooks/reading-scanned-documents]] — the sibling image cookbook: Noul + Box per item on a scan
- [[cookbooks/ocr-confidence-bounding-boxes]] — Box and crops used to check OCR values
- [[cookbooks/overview]] — the cookbook catalog
- [[ideas/patterns-emerging]] — community patterns for image evidence checks and find-and-locate

## Sources

- raw/docs/cookbooks__feeder_watch.md (https://docs.typesafe.ai/cookbooks/feeder_watch); re-fetched 2026-11-03 (published-cache download dropped, `ask_cached` no longer takes or stores the model; questions and numbers unchanged)
- raw/docs/cookbooks.md (https://docs.typesafe.ai/cookbooks) — section and level
- raw/docs/images.md (https://docs.typesafe.ai/images) — image limits and header
- raw/docs/primitives__box.md (https://docs.typesafe.ai/primitives/box) — Box contract