---
title: "Cookbook: Verifying LLM answers against their source"
type: cookbook
tags: [cookbook, verification, noul, rag, review-routing]
created: 2026-11-08
updated: 2026-11-11
confidence: high
sources:
  - raw/docs/cookbooks__verify_llm_outputs.md
  - raw/docs/cookbooks.md
  - raw/docs/primitives__noul.md
  - raw/docs/model-jaggedness.md
jev_version: "jev-1.14.0"
sdk_python: "0.7.5"   # cookbooks install cooksafe 0.3.0, which requires typesafe-sdk>=0.7.0,<0.8.0
summary: "Three Nouls in one request (answerable, declines, supported) become one risk number in code; answers at risk 0.2 or more go to review. Run on jev-1.14.0 over 1,000 MuSiQue answers from gpt-6-luna."
---

# Cookbook: Verifying LLM answers against their source

> **TL;DR** Send the question, the LLM's instruction, the retrieved paragraphs and the LLM's answer as one `state`, with three `Noul` questions in the same request: `answerable`, `declines`, `supported`. In code: if `declines >= 0.5`, risk = `answerable`; otherwise risk = `1 - supported`. Risk `>= 0.2` goes to review. On 1,000 `gpt-6-luna` answers (TypeSafe's run, `jev-1.14.0`, 2026-11-07) the check passed 727 answers, 95.7% of them right (88.5% across all), and held back 84 of the 115 wrong ones. It also held back 189 right answers, and 31 wrong answers passed.

## Goal

An LLM answers questions from paragraphs you retrieved for it, and most answers are right. The cookbook targets two kinds of mistake that, in its words, don't look "any different from a right answer":

- the LLM gives an answer the paragraphs never state, "filling a missing step with a guess";
- the LLM says the paragraphs don't answer the question when they do.

You end up with `check_answer()`. It takes a question, its paragraphs and an answer, and returns the three probabilities, the risk, and `pass` or `review`. The cookbook's diagram sends a `review` answer to "retry, a stronger model, or a person".

Upstream lists it under **LLMs and agents**, level **Beginner** (raw/docs/cookbooks.md).

## Inputs / state shape

**Data.** Questions from [MuSiQue](https://github.com/StonyBrookNLP/musique) (Trivedi et al., CC BY 4.0). Each joins two to four simple questions about Wikipedia into one, so the answer to an earlier part is needed to ask a later one. Each comes with 20 paragraphs, most of them on nearby topics that don't help. MuSiQue has two versions of each question: one where the paragraphs hold every needed fact, and one where a needed paragraph was removed, "though the remaining paragraphs still look as if they might" answer it. The cookbook's file `musique_questions.json` (downloaded from typesafe-public-examples, `cookbooks/verify_llm_outputs`) is a fixed random sample of 1,000 questions from MuSiQue's development set, one version each; about half can't be answered. The run prints `1,000 questions, 473 answerable from their paragraphs`.

Paragraphs are joined into one string as `## {title}\n{text}` blocks.

**The answers being checked** come from `gpt-6-luna` with this instruction, which allows a refusal (the source of the refusals the check reads):

```python
ANSWER_INSTRUCTION = (
    "Answer the question using only the paragraphs. Return only the shortest answer that fully "
    "answers the question. Do not explain your reasoning. If the paragraphs do not answer the "
    "question, say so."
)
```

To check your own LLM, the cookbook says to replace `answer_prompt()` and `ask_llm()`.

**The state sent to Jev**, one request per answer:

```python
def check_state(question: str, paragraphs: str, answer: str) -> dict:
    return {
        "request": question,
        "response_requirement": ANSWER_INSTRUCTION,
        "paragraphs": paragraphs,
        "candidate_response": answer,
    }
```

All 20 paragraphs go in, not just the relevant ones.

## Questions asked

Three `Noul` questions in the same request, verbatim:

```python
def noul(instructions: str, yes: str, no: str) -> Noul:
    return Noul(instructions=instructions, criteria=NoulCriteria(true=yes, false=no))


CHECK_QUESTIONS = {
    "answerable": noul(
        "Do the paragraphs state every fact needed to answer the question?",
        yes="each step from the question to its answer is stated in the paragraphs",
        no="at least one step from the question to its answer is not stated in the paragraphs",
    ),
    "declines": noul(
        "Does the candidate response decline to answer, or say the information is missing?",
        yes="the candidate says it cannot answer or that the paragraphs do not say",
        no="the candidate gives an answer",
    ),
    "supported": noul(
        "Following the paragraphs step by step from the question, do you arrive at the "
        "candidate's answer?",
        yes="the paragraphs connect the question to the candidate's answer through stated facts",
        no="the paragraphs do not connect the question to the candidate's answer",
    ),
}
```

| key | `type` | what `true` means |
|---|---|---|
| `answerable` | `noul` | every step from question to answer is stated in the paragraphs |
| `declines` | `noul` | the LLM said it cannot answer or the paragraphs don't say |
| `supported` | `noul` | the paragraphs connect the question to the LLM's answer through stated facts |

## Combining logic in code

The risk formula, verbatim from `check_answer()`:

- the LLM declined (`declines >= 0.5`): risk = `answerable` ("risky if the paragraphs do answer");
- the LLM answered: risk = `1 - supported` ("risky if the paragraphs don't lead there");
- `status` = `"review"` if risk `>= REVIEW_AT` (0.2), else `"pass"`.

Complete check, condensed from the cookbook's Setup and "Ask Jev three questions" blocks (the dataset download, the LLM answering step, the chart and the labelling step are left out; the check logic is verbatim; the last two lines are our placeholder call).

```python
import json
import os
import urllib.request
from pathlib import Path

from cooksafe import JsonCache
from typesafe_sdk import Noul, NoulCriteria, TypeSafeClient

TYPESAFE_MODEL = "jev-1.14.0"  # the page's model; "jev-latest" runs the newest, live
REVIEW_AT = 0.2  # a risk at or above this sends the answer to review

client = TypeSafeClient(
    api_key=os.environ.get("TYPESAFE_API_KEY", "cache-only"),  # keyless kernels replay the cache
    base_url=os.environ.get("TYPESAFE_ENDPOINT"),
    timeout=120.0,
)

EXAMPLES = (
    "https://raw.githubusercontent.com/typesafe-ai/typesafe-public-examples/main/"
    "cookbooks/verify_llm_outputs"
)


def download(name: str) -> Path:
    """Fetch one of this cookbook's files from typesafe-public-examples unless it is on disk."""
    path = Path(name)
    if not path.exists():
        urllib.request.urlretrieve(f"{EXAMPLES}/{name}", path)
    return path


CACHE = Path("verify_llm_outputs_cache.json")
# True fills CACHE with the published answers, so the run makes no requests. To run
# live, set it to False and delete CACHE; a CACHE already on disk is never overwritten.
DOWNLOAD_CACHE = True
if DOWNLOAD_CACHE:
    download(CACHE.name)
json_cache = JsonCache(CACHE, indent=None)

ANSWER_INSTRUCTION = (
    "Answer the question using only the paragraphs. Return only the shortest answer that fully "
    "answers the question. Do not explain your reasoning. If the paragraphs do not answer the "
    "question, say so."
)

def noul(instructions: str, yes: str, no: str) -> Noul:
    return Noul(instructions=instructions, criteria=NoulCriteria(true=yes, false=no))


CHECK_QUESTIONS = {
    "answerable": noul(
        "Do the paragraphs state every fact needed to answer the question?",
        yes="each step from the question to its answer is stated in the paragraphs",
        no="at least one step from the question to its answer is not stated in the paragraphs",
    ),
    "declines": noul(
        "Does the candidate response decline to answer, or say the information is missing?",
        yes="the candidate says it cannot answer or that the paragraphs do not say",
        no="the candidate gives an answer",
    ),
    "supported": noul(
        "Following the paragraphs step by step from the question, do you arrive at the "
        "candidate's answer?",
        yes="the paragraphs connect the question to the candidate's answer through stated facts",
        no="the paragraphs do not connect the question to the candidate's answer",
    ),
}


def check_state(question: str, paragraphs: str, answer: str) -> dict:
    return {
        "request": question,
        "response_requirement": ANSWER_INSTRUCTION,
        "paragraphs": paragraphs,
        "candidate_response": answer,
    }


def as_json(questions: dict) -> str:
    return json.dumps(
        {k: q.model_dump(mode="json", exclude_none=True) for k, q in questions.items()},
        sort_keys=True,
    )


@json_cache
def ask_jev(
    question: str, paragraphs: str, answer: str, questions_json: str, model: str
) -> dict:
    response = client.system_one(
        check_state(question, paragraphs, answer), json.loads(questions_json), model=model
    )
    return {
        "nouls": {k: response.answers[k].noul for k in sorted(response.answers)},
        "input_tokens": response.usage.input_tokens or 0,
    }


def check_answer(question: str, paragraphs: str, answer: str) -> dict:
    nouls = ask_jev(
        question, paragraphs, answer, as_json(CHECK_QUESTIONS), model=TYPESAFE_MODEL
    )["nouls"]
    if nouls["declines"] >= 0.5:
        risk = nouls["answerable"]  # it declined: risky if the paragraphs do answer
    else:
        risk = 1 - nouls["supported"]  # it answered: risky if the paragraphs don't lead there
    return {**nouls, "risk": risk, "status": "review" if risk >= REVIEW_AT else "pass"}


paragraphs = "## Title A\nText of paragraph A.\n\n## Title B\nText of paragraph B."
print(check_answer("Your question?", paragraphs, "The LLM's answer."))
```

The questions travel as JSON (`as_json`) so the serialized questions are part of the `JsonCache` key: editing a question forces a fresh call. Since the 2026-11-11 docs the model is an argument too (`model` on the Jev calls, `answering_model` on `ask_llm`, `recheck_model` on `recheck`), so changing a model also forces fresh calls. Install: `pip install openai ipython matplotlib 'cooksafe>=0.3.0,<0.4.0'` (since the 2026-11-11 capture; `'cooksafe>=0.2.0,<0.3.0'` in the 2026-11-08 one), then set `TYPESAFE_API_KEY` and `OPENAI_API_KEY`; cooksafe 0.3.0 (uploaded to PyPI 2026-11-10T23:44:13Z) requires `typesafe-sdk>=0.7.0,<0.8.0`, `msgspec>=0.18` and `lzstring`. With `DOWNLOAD_CACHE = True` (the default, since 2026-11-11) the code downloads the published `verify_llm_outputs_cache.json`, so a run makes no requests; to run live set it to `False` and delete the file (until then the cache was a local `json_cache.json`). The cookbook runs the 1,000 checks with `ThreadPoolExecutor(max_workers=4)`; its comment: "2,000 Jev calls; more workers hits the public rate limit". Rate limits: [[reference/rate-limits-and-errors]].

## Results the cookbook reports

Numbers from `jev-1.14.0` on 2026-11-07, checking answers `gpt-6-luna` wrote the same day (TypeSafe's run).

**Four example answers** (printed output):

| question | `gpt-6-luna` answer | answerable | declines | supported | risk | status |
|---|---|---|---|---|---|---|
| Where did the painter of The Mud Bath study? | The Slade School of Art. | 0.97 | 0.02 | 0.98 | 0.02 | PASS |
| Where is Assomada located? | The paragraphs do not say where Assomada is located. | 0.07 | 0.99 | 0.97 | 0.07 | PASS |
| What was the population density of Tammy Leitner's city of birth in 2009? | 182.5/km² | 0.12 | 0.04 | 0.08 | 0.92 | REVIEW |
| When did the father of Dahyabhai Patel die? | The paragraphs do not say when Dahyabhai Patel’s father died. | 0.86 | 0.81 | 0.09 | 0.86 | REVIEW |

In the cookbook's reading: Tammy Leitner was born in San Diego, but no paragraph gives San Diego's 2009 population density, so `supported` is low. The Patel question can be answered (one paragraph names Sardar Vallabhbhai Patel as the father, another gives his death date), so the refusal goes to review on `answerable` 0.86.

**Labels.** Each answer is labelled right or wrong. A separate Jev request with one `Noul` (`matches_reference`) compares the answer with MuSiQue's reference answer, without the paragraphs. A non-match (`< 0.5`) goes to `gpt-6.1-sol` (`reasoning_effort="high"`, JSON schema `right` / `wrong` / `unclear`), which reads the paragraphs, because "the reference is sometimes wrong about the paragraphs". Output: 739 matched the reference; of the other 261, `gpt-6.1-sol` labelled 146 right, 115 wrong and 0 unclear. The cookbook says the 146 are cases where MuSiQue counts a question answerable but the paragraphs skip a step, or the "unanswerable" version still has the answer in another paragraph.

**The measurement** (an answer passes when its risk is below the threshold):

| answers used | how many | right | wrong held back |
|---|---:|---:|---:|
| every answer | 1,000 (100%) | 88.5% | 0 of 115 |
| risk below 0.2 | 727 (73%) | 95.7% | 84 of 115 |
| risk below 0.3 | 825 (82%) | 94.2% | 67 of 115 |
| risk below 0.5 | 917 (92%) | 91.9% | 41 of 115 |
| risk below 0.7 | 965 (96%) | 90.3% | 21 of 115 |

At 0.2 the check holds back 273 answers: 84 wrong, 189 right, "so most of the review work goes to confirming answers that were fine". The chart (risk histogram, right vs wrong answers) shows right answers concentrated near zero risk and wrong answers "spread across the whole range", which is why the threshold has to be low.

The cookbook says the setup makes about 1,250 OpenAI calls and 2,000 Jev calls (code comments); it reports no cost or latency.

## Adapting it

- **Your LLM**: replace `answer_prompt()` and `ask_llm()`; put your own instruction in `response_requirement` so `declines` and `supported` read the answer against what the LLM was actually asked.
- **Your threshold**: the cookbook's advice is to run the labelling step on your own questions with reference answers and "pick the threshold from the table". 0.2 is tuned to this sample; it is not the docs' general Noul starting point (0.7 / 0.3 on a single Noul, [[concepts/noul]]).
- **Review route**: the diagram names retry, a stronger model or a person; which one is your call.
- **Your source**: anything that can be pasted as paragraphs works the same way (inferred; the cookbook tests only MuSiQue's Wikipedia paragraphs).
- **Playground**: `make_playground_link(check_state(...), CHECK_QUESTIONS)` opens the Patel refusal with its state and the three questions, to edit in the browser.

## Gotchas

- **Wrong answers that pass.** 31 wrong answers passed at 0.2. 18 answer a related question instead of the one asked (the cookbook's examples: "Leftover Salmon" when asked who is in the band, "the NFL" when asked what kind of thing the NFL is); the paragraphs do lead there, so `supported` comes back high. The other 13 are refusals of answerable questions where `answerable` also came back low, 0.10 to 0.19: "Jev missed the same connection between paragraphs that the LLM missed". The check is not independent of the LLM's own blind spots. Multi-hop questions are the indirection mode in [[concepts/jaggedness-jev-1-13]] (our link, not the cookbook's).
- **The branch is a hard cut on `declines`.** `declines >= 0.5` decides which Noul the risk reads; the other is ignored (the Tammy Leitner row reads `supported`, not `answerable`). A half-refusal near 0.5 flips branches (inferred).
- **Labels come from models.** "A stricter or looser re-check would change the 88.5%." The first pass of labelling is itself a Jev `Noul` (inferred concern: the check and part of its scoring share a model).
- **Selection on the sample.** TypeSafe picked these three questions over two other sets by trying them on the first 100 questions of this sample, so "the results on them are not a fair test".
- **Review load.** At 0.2, 189 of the 273 held-back answers were right; budget review for mostly-fine answers.
- **Rate limit.** More than 4 parallel Jev workers hit the public rate limit in TypeSafe's run.

## Related

- [[cookbooks/overview]] — the cookbook catalog
- [[cookbooks/citation-check]] — the other "check an LLM against its source" cookbook: one Choice per citation
- [[cookbooks/classifying-rag-passages]] — gating the passages before the LLM answers
- [[cookbooks/llm-guardrails]] — screening what goes into and out of an LLM
- [[patterns/fan-out]] — all three questions in one request
- [[patterns/companion-noul]] — one Noul deciding how to read another answer in the same request
- [[concepts/noul]] — the yes/no primitive and its thresholds
- [[guides/testing-and-evaluation]] — labelling and threshold-picking as an eval method

## Sources

- raw/docs/cookbooks__verify_llm_outputs.md (https://docs.typesafe.ai/cookbooks/verify_llm_outputs); re-fetched 2026-11-11 (install pin `cooksafe>=0.3.0,<0.4.0`, published `verify_llm_outputs_cache.json` downloaded via `DOWNLOAD_CACHE = True`, every cached call takes its model; questions, threshold, models, date and every number unchanged)
- raw/docs/cookbooks.md (https://docs.typesafe.ai/cookbooks.md) — section and level
- raw/docs/primitives__noul.md (https://docs.typesafe.ai/primitives/noul) — the 0.7 / 0.3 starting point
- raw/docs/model-jaggedness.md (https://docs.typesafe.ai/model-jaggedness) — indirection
