Agents: read the raw Markdown of this page, or start at llms.txt.
Cookbook: Infinite cardinality choice
TL;DR A
Choicetakes at most 255 options, so to pick one item out of thousands, narrow first: oneNoulper item ("is this the one they mean?") on a short description, sent in parallel chunks; a secondNoulround on a long description for the top 80; then oneChoiceover the top 20, whoseconfidencedecides "show one" (0.8 or more) or "show three". On 100 real "what was that film?" requests over 5,000 films (TypeSafe's run,jev-1.14.0, November 2026) the right film came first for 78%, against 33% when an embedding search built the shortlist. All 48 answers at 0.8 or more were right.
Goal
Someone describes a film they half-remember ("a man and a boy running from these sharp metal eyeball things with blades on the side"); the catalog holds it among 5,000 films, and its plot tells the same story in other words. One Choice could pick it from a list, but upstream notes it "takes at most 255 options, and only about forty when each option is a long plot summary". So the search runs in three steps, "and each step reads more about fewer films".
You end up with find(): it takes a request and returns the films ranked by probability, in three rounds of parallel calls. Upstream lists it under Search and matching, level Intermediate (raw/docs/cookbooks.md).
The 255 cap is the API's ("You can have a maximum of 255 options per Choice", raw/docs/api.md; the same in raw/docs/primitives__choice.md); the cookbook's code adds that "256 options is rejected outright". See Choice questions.
Inputs / state shape
State is the request text alone. Each question carries its own film, so the state stays short and every question is judged against it.
Catalog films.jsonl (downloaded from typesafe-public-examples, cookbooks/infinite_cardinality_choice): title, year and two cuts of the plot section of each film's Wikipedia article, fetched 2026-11-07. plot is "the first hundred or so words" (step 1 reads it); long_plot is "the first six hundred or so" (steps 2 and 3). The catalog holds the film behind every request; the other films, the ones a search must rule out, are those whose Wikipedia articles exist in the most languages.
Requests requests.jsonl: 100 from Microsoft's tip-of-the-tongue dataset, collected from the forum iRememberThisMovie.com, each with the film that answered it; a random sample of the 709 whose film has a plot section on Wikipedia. The worked example is request 388 (a mother and two kids haunted by ghosts that "turn out to be time travelers"); the film is The Diabolical (2015).
Questions asked
Steps 1 and 2: one Noul per film, no criteria, the film named in the instructions with its plot attached ("so Jev compares the request with the plot instead of recalling the film from its title"):
def proposition(film_id: str, plot: str = "plot") -> str:
return (
f"The film this person is trying to remember is {label(film_id)}. "
f"Its plot: {FILMS[film_id][plot]}"
)
Step 1 uses plot; step 2 the same function with "long_plot".
Step 3: one Choice, options keyed by film id, each option's description the title plus the long plot:
DECISION = "Which film is this person trying to remember?"
def decision_question(options: tuple[str, ...]) -> dict:
return {
"film": Choice(
instructions=DECISION,
criteria={key: f"{label(key)}: {FILMS[key]['long_plot']}" for key in options},
)
}
Why a Choice at the end: each Noul judges its film alone, so two films that both fit can both get 0.4 or more (here 0.60 and 0.40), and the top five "alone add up to more than 1". A Choice weighs the twenty against each other, so "its top probability says how far ahead of the rest the top film is".
Combining logic in code
The upstream steps put together into one script (setup condensed: the plotting imports and chart code dropped; the step functions are verbatim; decide() packs the cookbook's threshold printout into a function). It replays the published cache unless you set DOWNLOAD_CACHE = False and delete the cache file.
# pip install openai ipython matplotlib 'cooksafe>=0.3.0,<0.4.0'
import json
import os
import urllib.request
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from time import perf_counter
from cooksafe import JsonCache, make_playground_link
from typesafe_sdk import Choice, Noul, RetryPolicy, TypeSafeClient
TYPESAFE_MODEL = "jev-1.14.0" # the page's model; "jev-latest" runs the newest, live
CHOICE_LIMIT = 255 # enforced by the API; 256 options is rejected outright
CHUNK = 300 # films per filter call: ~150 tokens each keeps a call well under 64k
LONG_CHUNK = 40 # films per re-read call: ~700 tokens each, ~28k a call
FIRST_CUT = 80 # films read again with their long plot
CANDIDATES = 20 # films the Choice question sees
THRESHOLD = 0.8 # show one film at or above this Choice confidence
client = TypeSafeClient(
api_key=os.environ.get("TYPESAFE_API_KEY", "cache-only"), # keyless kernels replay the cache
base_url=os.environ.get("TYPESAFE_ENDPOINT"),
timeout=300.0, # 300 questions is a big request
# a request this size occasionally draws an overload response, so retries are not optional
retry=RetryPolicy(max_retries=5, backoff_initial=1.0, backoff_max=30.0),
)
EXAMPLES = (
"https://raw.githubusercontent.com/typesafe-ai/typesafe-public-examples/main/"
"cookbooks/infinite_cardinality_choice"
)
def download(name: str) -> Path:
"""Fetch one of this cookbook's files from typesafe-public-examples unless it is on disk."""
path = Path(name)
if not path.exists():
path.parent.mkdir(parents=True, exist_ok=True)
urllib.request.urlretrieve(f"{EXAMPLES}/{name}", path)
return path
CACHE = Path("infinite_cardinality_choice_cache.json")
# True fills CACHE with the published answers, so the run makes no requests. To run
# live, set it to False and delete CACHE; a CACHE already on disk is never overwritten.
DOWNLOAD_CACHE = True
if DOWNLOAD_CACHE:
download(CACHE.name)
json_cache = JsonCache(CACHE)
FILMS = {f["id"]: f for f in map(json.loads, download("films.jsonl").read_text().splitlines())}
REQUESTS = [json.loads(line) for line in download("requests.jsonl").read_text().splitlines()]
def label(film_id: str) -> str:
film = FILMS[film_id]
return f"{film['title']} ({film['year']})" if film["year"] else film["title"]
def proposition(film_id: str, plot: str = "plot") -> str:
return (
f"The film this person is trying to remember is {label(film_id)}. "
f"Its plot: {FILMS[film_id][plot]}"
)
def as_json(questions: dict) -> str:
"""The questions as API JSON, so the cache key changes when their wording does."""
return json.dumps(
{key: q.model_dump(mode="json", exclude_none=True) for key, q in questions.items()}
)
IDS = sorted(FILMS)
CHUNKS = [
as_json({i: Noul(instructions=proposition(i)) for i in IDS[start : start + CHUNK]})
for start in range(0, len(IDS), CHUNK)
]
# Step 1: one Noul per film on the short plot, every chunk sent at once
@json_cache
def filter_chunk(text: str, questions_json: str, model: str) -> dict:
started = perf_counter()
response = client.system_one(
state=text, questions=json.loads(questions_json), model=model
)
return {
"p": {key: answer.noul for key, answer in response.answers.items()},
"seconds": round(perf_counter() - started, 2),
}
def filter_films(text: str) -> dict:
"""Every chunk at once; one probability per film."""
with ThreadPoolExecutor(max_workers=len(CHUNKS)) as pool:
parts = list(
pool.map(lambda chunk: filter_chunk(text, chunk, model=TYPESAFE_MODEL), CHUNKS)
)
return {key: p for part in parts for key, p in part["p"].items()}
def by_probability(request: dict) -> list[str]:
p = filter_films(request["text"])
return sorted(p, key=lambda key: (-p[key], key))
# Step 2: the top 80 again, with the long plot
def reread(request: dict) -> dict:
"""The filter's top films, asked again with the long plot; one probability per film."""
first = by_probability(request)[:FIRST_CUT]
chunks = [
as_json({i: Noul(instructions=proposition(i, "long_plot"))
for i in first[start : start + LONG_CHUNK]})
for start in range(0, FIRST_CUT, LONG_CHUNK)
]
with ThreadPoolExecutor(max_workers=len(chunks)) as pool:
parts = list(
pool.map(
lambda chunk: filter_chunk(request["text"], chunk, model=TYPESAFE_MODEL), chunks
)
)
return {key: p for part in parts for key, p in part["p"].items()}
def candidates(request: dict) -> tuple[str, ...]:
p = reread(request)
return tuple(sorted(p, key=lambda key: (-p[key], key))[:CANDIDATES])
# Step 3: one Choice over the top 20
DECISION = "Which film is this person trying to remember?"
def decision_question(options: tuple[str, ...]) -> dict:
return {
"film": Choice(
instructions=DECISION,
criteria={key: f"{label(key)}: {FILMS[key]['long_plot']}" for key in options},
)
}
@json_cache
def rank(text: str, questions_json: str, model: str) -> dict:
response = client.system_one(
state=text, questions=json.loads(questions_json), model=model
)
answer = response.answers["film"]
return {"film": answer.choice, "confidence": answer.confidence,
"probabilities": dict(answer.probabilities)}
def find(request: dict, options: tuple[str, ...] | None = None) -> dict:
"""Rank a shortlist with one Choice; by default, the top twenty on the long plots."""
options = candidates(request) if options is None else options
ranked = rank(request["text"], as_json(decision_question(options)), model=TYPESAFE_MODEL)
p = ranked["probabilities"]
return {"film": ranked["film"], "order": sorted(p, key=lambda key: (-p[key], key)),
"confidence": ranked["confidence"], "probabilities": p}
def decide(result: dict) -> list[str]:
"""One film when the Choice is sure enough, otherwise the top three."""
return result["order"][:1] if result["confidence"] >= THRESHOLD else result["order"][:3]
example = next(r for r in REQUESTS if r["id"] == "388")
result = find(example)
print([label(key) for key in decide(result)])
print(make_playground_link(example["text"], decision_question(candidates(example))))
Per search that is 17 filter calls of up to 300 Nouls, 2 re-read calls of 40 and 1 Choice call: 20 requests and 5,081 questions (counted from the constants, inferred). Upstream's sizing: about 150 tokens per film in step 1, about 700 in step 2 ("~28k a call"), to stay under the 64k per-request context (Models, aliases, pricing, rate limits, context: "64k per-request / 32k per-question"). The chunks don't depend on each other, so they all go out at once, and "the wait stays about the same as the catalog grows, as long as your rate limit allows that many calls at a time". The measurement runs WORKERS = 4 searches side by side.
Results the cookbook reports
TypeSafe's run: "Numbers below came from jev-1.14.0, run in November 2026" (no exact day; the plots were fetched 2026-11-07).
The worked example (request 388). Step 1: "5,000 answers from 17 calls sent together; the slowest took 0.4s". The filter ranks Ghostbusters: Afterlife first at 0.64 (mother, two kids, haunted house, money trouble); The Diabolical is at position 6, because its time-traveller twist "comes after the first hundred words, so the filter never sees it". Step 2 on long plots: The Diabolical 0.60, Ghostbusters: Afterlife 0.40, The Others 0.30, Thirteen Ghosts 0.26, Knowing 0.17. Step 3: the Choice puts 1.00 on The Diabolical.
Across 100 requests, the filter's top 80 hold the right film for 93; after the re-read the top 20 hold it for 88, against 82 for the filter's own top 20. Three shortlists go to the same Choice:
| embeddings | no re-read | all 3 steps | |
|---|---|---|---|
| reaches the Choice | 35% | 82% | 88% |
| film in the top 1 | 33% | 73% | 78% |
| film in the top 3 | 34% | 76% | 82% |
embeddings = OpenAI text-embedding-3-small over each film's label: plot (short plot) and the request, twenty nearest; upstream calls this "the standard first try". no re-read skips step 2. "The Choice question can only rank the films it is given."
Show one film or a list. How often the top answer was right, by its probability:
| top answer | long-plot Nouls | Choice question |
|---|---|---|
| under 0.5 | 30/49 right | 9/30 right |
| 0.5 to 0.8 | 33/40 right | 21/22 right |
| 0.8 to 0.95 | 11/11 right | 18/18 right |
| 0.95 and up | 0/0 right | 30/30 right |
"Choice at 0.8 or more: 48 requests, right for 48"; "below 0.8: 52 requests, the film in the top 3 for 34". The re-read's probability "can't sort requests into sure and unsure": none reaches 0.95, only 11 reach 0.8, and its answers under 0.5 are still right 30 times in 49. Upstream's rule: at 0.8 or more show the one film; below, show the top three.
The playground link holds step 3 only (the request and the Choice over the re-read's twenty), because "the playground can't comfortably show thousands of questions".
Adapting it
- Upstream: "give each item a short and a long description and rewrite
FILMS,proposition()anddecision_question(). Nothing else mentions films." - Re-tune
CHUNKandLONG_CHUNKto your descriptions' token length so each call stays under 64k; keepCANDIDATESwell below 255 when options are long. - Re-measure
FIRST_CUTrecall on your data first: the Choice can't recover an item the filter dropped. - Re-pick
THRESHOLDfrom your own band table; 0.8 is this run's.
Gotchas
- Recall ceiling. 7 of 100 requests lose the right film in step 1 and 12 never reach the Choice; no Choice tuning fixes that.
- Short descriptions miss late details. "People remember scenes from anywhere in a film", hence the long-plot re-read.
- Don't threshold the Noul scores for the final call. They are independent per film (several can be high at once) and, on this run, badly separated; threshold the Choice.
- Prose says probability, code reads
confidence. Upstream's text says "decide with the Choice question's top probability", but its threshold code comparesf["confidence"] >= THRESHOLD; the band table bins top probability. On this run both give 48 requests (18 + 30 in the table). The two are different numbers (Confidence vs probability); pick one and calibrate it. - Volume. About 5,000 Noul questions per search (inferred); upstream gives no cost for this cookbook, so price it with Models, aliases, pricing, rate limits, context before production, and check your rate limit can take the parallel burst ("as long as your rate limit allows").
- Retries are part of the design. Upstream sets a 300 s timeout and
RetryPolicy(max_retries=5, ...)because "a request this size occasionally draws an overload response". - Sample and baseline. 100 requests; the distractors are the most-translated films, and the embedding baseline is one model on short plots (inferred: a different catalog or a stronger retriever may move both columns).
- Published cache. The default run replays TypeSafe's answers; the model is part of each cached call's key, so changing
TYPESAFE_MODEL(or"jev-latest") calls live.
Related
- Cookbooks overview — the cookbook catalog
- Choice questions — the 255-option limit and how Choice probabilities read
- Noul (yes/no) questions — the per-item yes/no question
- Screen with a Noul first — screening with a Noul before the costlier step
- Speculative fan-out — many questions per request, many requests at once
- Routing by probability — acting on the top Choice probability
- Cookbook: Re-ranking — one Noul per query-candidate pair over a BM25 shortlist
- Cookbook: Line-by-line search — one Choice over 218 line ids inside one document
- Cookbook: Skill suggestion — rank then re-check, over 194 skills
- Cookbook: Hierarchical classification — another way past a large option set: walk a taxonomy
- Models, aliases, pricing, rate limits, context — context limits, rate limits, prices
Sources
- raw/docs/cookbooks__infinite_cardinality_choice.md (https://docs.typesafe.ai/cookbooks/infinite_cardinality_choice)
- raw/docs/cookbooks.md (https://docs.typesafe.ai/cookbooks.md) — section and level
- raw/docs/api.md (https://docs.typesafe.ai/api) — "a maximum of 255 options per Choice"
- raw/docs/primitives__choice.md (https://docs.typesafe.ai/primitives/choice) — "up to 255 options"
- raw/docs/models.md (https://docs.typesafe.ai/models) — 64k per-request / 32k per-question context