$jevwiki.ai#an LLM wiki about Jev, written for agents rather than people

Agents: read the raw Markdown of this page, or start at llms.txt.

~/wiki/cookbooks

Cookbook: Feeder-cam visit log

[ cookbook ][ updated 2026-11-03 ][ confidence high ][ jev-1.14.0 ][ python sdk 0.7.5 ]#cookbook · images · box · choice · noul · video

TL;DR Send each video frame as an Image with five questions in one request: a Noul (is there a bird?), a Choice (none / one / several), a Box (the most prominent bird) and two species Choices (leftmost, rightmost bird). One Box returns one box, so when two species share the feeder a second request asks for a Box per species by name; boxes that overlap by 0.6 or more are merged. Plain code turns the per-frame answers into visits. On a 101-second clip (jev-1.14.0, 2026-10-15): 178 requests, bird_present agreed with hand labels on 101/101 frames, the right species came back for 132 of 133 labeled birds, and both long visits matched the hand-label log to the second. Images and Box are experimental and need the TypeSafe-Danger-Zone-Images: true header.

Goal

A camera on a bird feeder, a front door or a street records hours of footage. The cookbook sends Jev one frame per second, asks the same questions about each, and uses ordinary code to turn the answers into a visit log per species: who came, from when to when, for how long (raw/docs/cookbooks__feeder_watch.md).

Upstream lists it under "Image processing" ("Find things in photos and scans, and check what was read from them"), level Intermediate, next to Cookbook: Reading a scanned vaccination record and Cookbook: OCR confidence and bounding boxes (raw/docs/cookbooks.md; Cookbooks overview). It is one of the first official cookbooks built on the experimental image input (Images and assets (experimental)) and Box question (Box questions (experimental)).

When to use it: a fixed camera, a closed list of things you expect to see, and a per-frame yes/no plus "which one" and "where" that code can aggregate over time. When not to: a flock or several of the same species in one frame (a named Box cannot tell two cardinals apart), counting beyond "several", or frames below the image size floor. The cookbook's own advice for flocks is a zero-shot detector (it names Meta's SAM 3) to find every bird, then Jev to name the species of each crop.

Inputs / state shape

Each request's state is one frame under the key frame, sent as an SDK Image; every question refers to `frame`:

state={"frame": Image(FRAMES / frame)}

The footage is "Busy Bird Feeder", a 101-second public-domain clip filmed by Courtney Celley of the U.S. Fish and Wildlife Service. For the first minute a male cardinal and a downy woodpecker share the feeder; then the male cardinal leaves and a female cardinal feeds from behind the feeder, mostly out of sight. TypeSafe cut one frame a second at 1024 × 576 and hand-labeled each frame with every bird in full view: 11 frames with no bird in full view, 27 with one, 63 with two. Twenty-two frames are marked partial (the female cardinal cut off or mostly hidden). Frames (usfws_frames.zip) and labels (usfws_labels.json) are in the typesafe-ai/typesafe-public-examples repo. Every response is saved in a local json_cache.json, so a second run makes no requests; until the 2026-11-03 docs the cookbook also published that cache there, to replay the published numbers without a key. A full run sends about 180 requests.

Numbers below came from jev-1.14.0 on 2026-10-15 (the cookbook passes model="jev-1.14.0" on every call, commented "Box questions need Jev 1.14"; jev-latest has pointed to jev-1.14.0 since 2026-10-24, Models, aliases, pricing, rate limits, context).

Setup, verbatim except the omitted plotting code:

import json
import os
from pathlib import Path
from time import perf_counter

from cooksafe import JsonCache
from typesafe_sdk import Box, Choice, Image, Noul, SystemOneResponse, TypeSafeClient

TYPESAFE_MODEL = "jev-1.14.0"  # Box questions need Jev 1.14
IMAGE_HEADERS = {"TypeSafe-Danger-Zone-Images": "true"}  # images and Box are opt-in

client = TypeSafeClient(
    api_key=os.environ.get("TYPESAFE_API_KEY", "cache-only"),  # keyless runs replay the cache
    timeout=60.0,
)
json_cache = JsonCache(Path("json_cache.json"))  # every response, keyed by frame and questions

Install upstream: pip install 'typesafe-sdk>=0.7.5,<0.8.0' pillow matplotlib ipython (or the same with uv add). That line does not install cooksafe, which the script imports; add 'cooksafe>=0.2.0,<0.3.0' as the sibling image cookbook does (inferred fix; Cookbook: Reading a scanned vaccination record).

Questions asked

Five questions, one request per frame, so the image is sent once:

Key type instructions criteria
bird_present noul "Is there at least one bird in frame?" none
how_many choice "How many birds are in frame?" none "No bird at all."; one "Exactly one bird."; several "Two or more birds."
bird box "The most prominent bird in frame" none (Box takes none)
species_left choice "What species is the leftmost bird in frame?" SPECIES (ten species plus other)
species_right choice "What species is the rightmost bird in frame?" SPECIES

With one bird in the frame both species answers describe it; with two, each describes one bird. Each species option describes what the bird looks like on a feeder camera, not what a field guide says: the grackle option warns it can fill the frame so only its head shows.

SPECIES = {
    "blue_jay": "A blue bird with a crest, a pale grey-white face and chest, a black necklace, "
    "and black bars on the blue wings and tail.",
    "cardinal_male": "A bright red bird with a pointed crest, a black mask, and an orange beak.",
    "cardinal_female": "A warm brown or buff bird with a pointed crest, reddish tinges on the "
    "wings, tail and crest, and an orange beak.",
    "grackle": "A common grackle: a glossy black bird with a pale yellow eye, a long heavy dark "
    "beak, and an iridescent blue, purple or bronze sheen on the head and body. On a close-up "
    "feeder camera it often fills the frame, so only the glossy dark head, pale eye and beak "
    "may be visible.",
    "starling": "A European starling: a chunky short-tailed dark bird with glossy plumage "
    "speckled with pale spots, a dark eye, and a pointed yellow or dark beak. Juveniles are "
    "plain grey-brown with a dark beak.",
    "mourning_dove": "A plump grey-brown dove with a small head, a long pointed tail, and black "
    "spots on the wings.",
    "house_sparrow": "A small streaky brown sparrow. Males have a grey crown and a black bib.",
    "house_finch": "A small streaky brown finch. Males have a rosy red head and chest.",
    "red_winged_blackbird": "A glossy black bird with a red shoulder patch edged in yellow.",
    "downy_woodpecker": "A small black-and-white woodpecker with a white stripe down the back "
    "and spotted wings. Males have a red patch on the back of the head.",
    "other": "A bird that matches none of the other descriptions.",
}

QUESTIONS = {
    "bird_present": Noul(instructions="Is there at least one bird in `frame`?"),
    "how_many": Choice(
        instructions="How many birds are in `frame`?",
        criteria={
            "none": "No bird at all.",
            "one": "Exactly one bird.",
            "several": "Two or more birds.",
        },
    ),
    "bird": Box(instructions="The most prominent bird in `frame`"),
    "species_left": Choice(
        instructions="What species is the leftmost bird in `frame`?", criteria=SPECIES
    ),
    "species_right": Choice(
        instructions="What species is the rightmost bird in `frame`?", criteria=SPECIES
    ),
}

The call itself (verbatim): the questions are serialized to JSON so the cache can key on them, and the images header goes on every request with extra_headers:

@json_cache
def ask_cached(frame: str, questions_json: str) -> dict:
    started = perf_counter()
    response = client.system_one(
        state={"frame": Image(FRAMES / frame)},
        questions=json.loads(questions_json),
        model=TYPESAFE_MODEL,
        extra_headers=IMAGE_HEADERS,
    )
    seconds = round(perf_counter() - started, 2)
    return {"response": response.model_dump(mode="json", exclude={"model"}), "seconds": seconds}


def ask(path: Path, questions: dict):
    as_json = json.dumps(
        {key: q.model_dump(mode="json", exclude_none=True) for key, q in questions.items()},
        sort_keys=True,
    )
    cached = ask_cached(path.name, as_json)
    # The cache leaves the model out; the response type needs one, so use the requested model.
    response = SystemOneResponse.model_validate({**cached["response"], "model": TYPESAFE_MODEL})
    return response, cached["seconds"]

Until the 2026-11-03 docs ask_cached took the model as its first argument and cached the whole response, model included; it now keys on frame and questions only and leaves model out of the cached response.

Second request: a Box per species by name. Asking for "the leftmost bird" and "the rightmost bird" as two Boxes failed: on frame u010 both boxes landed on the cardinal (confidence 0.76 and 0.32). Naming the species the first request found worked; where a name alone could fit another bird, the phrase adds what it looks like:

BOX_TARGETS = {
    "blue_jay": "the blue jay",
    "cardinal_male": "the male northern cardinal (the bright red bird)",
    "cardinal_female": "the female northern cardinal (the brown bird with an orange beak)",
    "grackle": "the common grackle (the glossy black bird)",
    "starling": "the European starling (the speckled dark bird)",
    "mourning_dove": "the mourning dove",
    "house_sparrow": "the house sparrow",
    "house_finch": "the house finch",
    "red_winged_blackbird": "the red-winged blackbird",
    "downy_woodpecker": "the downy woodpecker (the black and white bird)",
    "other": "the bird that is none of the listed species",
}


def box_questions(species: list[str]) -> dict:
    return {sp: Box(instructions=f"{BOX_TARGETS[sp]} in `frame`") for sp in species}

On u010 the named boxes separated: cardinal_male left 0.13 right 0.42 (confidence 0.84), downy_woodpecker left 0.59 right 0.73 (confidence 0.36).

Combining logic in code

Three settings decide the per-frame answer, all in code: PRESENT_AT = 0.5, SPECIES_AT = 0.5, OVERLAP_AT = 0.6. A Box always returns a box, even on an empty frame or for a species that is barely there, so the Noul decides whether a box is used, and two named boxes overlapping by 0.6 or more (intersection over union) count as one bird found twice; only the higher-confidence box is kept.

OVERLAP_AT = 0.6  # two named boxes overlapping this much are one bird found twice


def overlap(a: dict, b: dict) -> float:
    """Intersection over union of two boxes given as edge fractions."""
    w = max(0.0, min(a["right"], b["right"]) - max(a["left"], b["left"]))
    h = max(0.0, min(a["bottom"], b["bottom"]) - max(a["top"], b["top"]))
    area = lambda e: (e["right"] - e["left"]) * (e["bottom"] - e["top"])  # noqa: E731
    union = area(a) + area(b) - w * h
    return w * h / union if union > 0 else 0.0


def dedupe(birds: list[dict]) -> list[dict]:
    kept: list[dict] = []
    for bird in sorted(birds, key=lambda b: -b["box_confidence"]):
        if all(overlap(bird["box"], k["box"]) < OVERLAP_AT for k in kept):
            kept.append(bird)
    return kept


PRESENT_AT = 0.5  # below this, the frame is empty and the other answers are ignored
SPECIES_AT = 0.5  # a species answer below this is too unsure to name a box


def ask_frame(path: Path) -> dict:
    first, seconds = ask(path, QUESTIONS)
    result = {
        "frame": path.name,
        "bird_present": first.nouls["bird_present"].noul,
        "how_many": first.choices["how_many"].choice,
        "birds": [],
        "requests": 1,
        "seconds": seconds,
    }
    # the species named by the leftmost and rightmost answers, each with its best probability
    named: dict[str, float] = {}
    sure_sides = 0
    for key in ("species_left", "species_right"):
        answer = first.choices[key]
        p = answer.probabilities[answer.choice]
        if p >= SPECIES_AT:
            named[answer.choice] = max(p, named.get(answer.choice, 0.0))
            sure_sides += 1
    if result["bird_present"] < PRESENT_AT or not named:
        return result

    # with several birds, the `bird` box only fits when both sides name the same species
    if result["how_many"] == "several" and (len(named) == 2 or sure_sides < 2):
        second, more_seconds = ask(path, box_questions(list(named)))
        result["requests"] = 2
        result["seconds"] += more_seconds
        boxes = {sp: second.boxes[sp] for sp in named}
    else:
        top = max(named, key=named.get)
        named, boxes = {top: named[top]}, {top: first.boxes["bird"]}

    birds = [
        {"species": sp, "p": p, "box": boxes[sp].box.model_dump(), "box_confidence": boxes[sp].confidence}
        for sp, p in named.items()
    ]
    result["birds"] = dedupe(birds)
    return result

The species probability used is probabilities[choice] (the top option's probability), not the Choice confidence field (Confidence vs probability). Frames are independent, so the batch run uses ThreadPoolExecutor(max_workers=4) and pool.map (results stay in frame order); on a live camera, upstream says to send one frame at a time, each as soon as the previous answer is back.

Visits are built in frame order, one species at a time (two birds at once make two overlapping visits). A frame counts for a species at species_at=0.6; a visit stays open across gaps of up to max_gap=2 frames; a visit needs min_frames=2 frames naming its species, so a one-frame misread does not become a visit. Jev never sees these settings; re-running visits() on saved results costs no requests.

def visits(results: list[dict], species_at=0.6, max_gap=2, min_frames=2) -> list[dict]:
    done, open_visits, last_seen = [], {}, {}
    for second, result in enumerate(results):  # one frame per second
        seen = {b["species"] for b in result["birds"] if b["p"] >= species_at}
        for sp in list(open_visits):
            if sp not in seen and second - last_seen[sp] > max_gap:
                done.append(open_visits.pop(sp))
        for sp in sorted(seen):
            if sp in open_visits:
                open_visits[sp]["end"] = second
                open_visits[sp]["frames"] += 1
            else:
                open_visits[sp] = {"species": sp, "start": second, "end": second, "frames": 1}
            last_seen[sp] = second
    done.extend(open_visits.values())
    kept = [v for v in done if v["frames"] >= min_frames]
    return sorted(kept, key=lambda v: (v["start"], v["species"]))

Results / what the cookbook reports

All from jev-1.14.0, 2026-10-15 (raw/docs/cookbooks__feeder_watch.md).

Three sample frames, first request:

Frame Label Seconds bird_present how_many bird box confidence species_left / species_right
u010 cardinal_male, downy_woodpecker 0.35 0.98 several 1.00 0.71 (covers only the cardinal) cardinal_male 1.00 / downy_woodpecker 1.00
u090 downy_woodpecker 0.29 0.95 one 1.00 0.85 downy_woodpecker 0.99 / 0.99
u097 empty 0.28 0.07 none 0.99 0.14 (a box anyway) other 0.38 / other 0.61

On the empty frame the species questions still named an option (other); the low bird_present is what tells code to ignore them.

Full run: 101 frames, 178 requests; 77 frames needed the second request, 66 ended with two boxes (the rest had only one sure species answer, or dedupe() merged the boxes); median 0.54 s per frame.

Visit log:

Species From the answers From the hand labels
cardinal_male 0:00 to 1:02 (63 s) 0:00 to 1:02 (63 s)
downy_woodpecker 0:00 to 1:29 (90 s) 0:00 to 1:29 (90 s)
cardinal_male 1:12 to 1:15 (4 s), wrong: the female cardinal behind the feeder, misnamed when only her red beak showed none

In every run TypeSafe made of the clip the two long visits came out the same; the short visits changed from run to run. With min_frames=1, four single-frame sightings appear: cardinal_female at 1:04 and 1:08 (real), other at 1:16, cardinal_male at 1:34 (a misread). A higher min_frames keeps out more misreads and drops more birds that show for a second or two.

Against the labels: bird_present agreed on 101/101 frames (partial frames count as having a bird); 132/133 labeled birds were returned (partial frames left out of this check). The one miss is u063 (1:02): the woodpecker, sharing the frame with the male cardinal, came back as only cardinal_male. The hardest presence frames were 1:33 and 1:34, where only part of a cardinal shows; Jev scored both about 0.8 and both are labeled partial. One failure dedupe() does not catch: at 1:15 Jev named a male cardinal from the female's beak, and the male-cardinal box landed on the woodpecker, overlapping the woodpecker's own box by less than 0.6, so both stayed. Upstream suggests a per-box check: crop to the box and ask a Noul whether the bird in the crop is that species.

Live: a small web app on the same questions (webcam or video file, boxes drawn, running visit log) played a stretch of the Ostdrossel feeder cam, which the questions were not tuned on, at about two answers a second (video demo; no accuracy figure given).

Adapting it to a new domain

Gotchas

Sources