--- title: "Running Jev in production: versioning, caching, retries, monitoring and fallbacks" type: guide source_tier: mixed tags: [production, operations, versioning, caching, monitoring, reliability] created: 2026-09-23 updated: 2026-09-23 confidence: medium sources: - raw/docs/api.md - raw/docs/models.md - raw/docs/confidence.md - raw/docs/concepts__how-to-build-with-system-one.md - raw/docs/model-jaggedness__jev-1.13.md - raw/docs/patterns__confidence-routing.md - raw/docs/cookbooks__consistency_choice_cookbook.md - raw/docs/cookbooks__consistency_noul_cookbook.md - raw/docs/cookbooks__sde_cascade.md - raw/docs/cookbooks__parallel_questions.md - raw/docs/cookbooks__entity_alignment.md - raw/docs/cookbooks__autoresearch_feature_discovery.md - raw/docs/sdk__python__api__retries.md - raw/docs/sdk__python__api__exceptions.md - raw/docs/sdk__python__api__constants.md - raw/docs/sdk__python__usage.md - raw/docs/sdk__javascript__api__interfaces__RetryPolicy.md - raw/docs/legal.md - raw/github/typesafe-sdk-python/src/typesafe_sdk/_core/transport.py - raw/github/typesafe-sdk-python/src/typesafe_sdk/_core/constants.py - raw/github/typesafe-sdk-js/src/client.ts - raw/site/typesafe-ai-legal_mca.txt - raw/community/latent-space-jev-diogo-almeida.md - raw/community/archerhume-jevs-architecture-unmasked.md - raw/community/tjklug-com-posts-typesafe-jev-slopcheck.md - raw/community/dev-to-chorylee-i-ran-300-real-jev-calls-on-a-simulated-robot-fleet-the-bill-was-000737-45.md - raw/community/vercel-com-kb-guide-typesafe-jev-and-ai-sdk.md - raw/x-repos/anessbelbati__jev-rerank-bench.md - raw/x-repos/shiftynick__jev-axi.md - raw/x-repos/TPAteeq__tocsin.md - raw/x-repos/seb4ez__jevguard.md - raw/x-repos/mizchi__jev-test-filter.md - raw/x-repos/kierandotai__jev-scout.md - raw/x-repos/shitianfang__wakegate.md - raw/x-repos/stas4000__jev-papers.md - raw/x-repos/clownware__bouncer.md - raw/x-repos/JoacoMarc__jev-harness-router.md - raw/x-repos/rlaope__jeval.md - wiki/reference/rate-limits-and-errors.md - wiki/reference/python-sdk-responses.md - wiki/reference/legal-and-data.md - wiki/guides/testing-and-evaluation.md - wiki/syntheses/faq.md jev_version: "jev-1.13.0" sdk_python: "0.7.1" sdk_js: "0.6.0" summary: "Pre-launch checklist for a Jev pipeline: pin and version every decision input, cache by exact request, retry transport not doubt, gate on confidence, log for replay, monitor drift, plan fallbacks." --- # Running Jev in production: versioning, caching, retries, monitoring and fallbacks > **TL;DR** Pin `model` to a versioned ID (`jev-1.13.0`); version questions, option lists and thresholds together. Cache on exact state + question spec + model ID. Let the SDK retry transport failures (`408`, `429`, `5xx`, connection, timeout). Act only on answers from the pinned model; escalate low-confidence answers, never re-ask. Log answers with question version, returned `model`, probabilities, request ID; replay from the log. Monitor latency, errors, drift, cost. Plan fallbacks: no uptime SLA. Unsourced advice is marked "(inferred)". Labelled calibration set: [[guides/testing-and-evaluation]]. ## 1. Version everything that shapes a decision One version string per decision covers: - [ ] `instructions` text, including structured-instruction fields. - [ ] `criteria`: Choice option keys, their descriptions **and their order**; Score levels and their order; Noul `true`/`false` descriptions. - [ ] Question type: "Don't carry a threshold tuned on a Noul over to a Choice" (raw/docs/model-jaggedness__jev-1.13.md). - [ ] Thresholds and composition weights in your code. - [ ] The state builder: which fields, in what order, serialised how. - [ ] `model`: a versioned ID, not an alias. Why: - **A reworded question is a new instrument.** The same question as a Noul and as a yes/no Choice gave `noul` 0.22 against `yes` 0.01; a negated Noul pair summed to 1.19 (raw/docs/model-jaggedness__jev-1.13.md). Thresholds tuned on version N do not carry to N+1 (inferred). - **Options interact.** Archer Hume: adding an option shifted the odds between existing ones; reversing option order moved a classification from ~0.84–0.89 to 0.93–0.96 (community, `unverified`). - **State order matters too** (docs silent). TJ Klug: a Noul read 0.33 with `goal` after a kilobyte of diff, 0.93 with `goal` first. Reversing 30 passages in `state` changed Jev Choice's top pick on 24.7% of 1,617 queries (anessbelbati/jev-rerank-bench). Community, `unverified`. - **Aliases move.** "An alias moves when a new release ships"; with tuned thresholds, "pin that version's ID" (raw/docs/models.md). `jev-latest` is the SDK default (raw/docs/sdk__python__api__constants.md). - **Question IDs are free:** "not sent to the underlying model" (raw/docs/api.md). `GET /v1/models` "currently lists the aliases", not what they resolve to; read that from each response's `model`. ## 2. Caching **Key on the full request:** 1. The exact serialised `state`. 2. The full question spec (`type`, `instructions`, `criteria`) in the order you send it. 3. The pinned model ID. Do not sort keys in `criteria` or `state`: order is part of the instrument (§1); a cache that canonicalises key order (JevGuard does) can merge requests Jev answers differently (inferred). The cookbooks key on a digest of "everything that shapes the prompt/rubric" and keep the returned model "because an alias can resolve to a different version later" (raw/docs/cookbooks__consistency_noul_cookbook.md). **Do not cache:** errors and `retry_later` outcomes; a response whose `model` differs from your pin; requests carrying a nonce (the consistency cookbooks add a `uid` to force fresh draws); calibration runs; answers depending on a fact not in `state`, such as "now" (put the date in `state`, as [[cookbooks/date-extraction]] does). **TTLs.** None in the docs. Community: jev-axi reuses answers up to 24 h while `jev-latest` resolves to the same version; tocsin replaces verdicts when a newer model answers. Pinned, an exact-input entry is valid until the question version or pin changes; behind an alias it can silently belong to an older model: pin rather than shorten the TTL (inferred). ## 3. Retries and errors | Kind | Examples | Action | |---|---|---| | Transport failure | `408`, `429`, `5xx` including `529 Overloaded`, connection error, timeout | Retry with backoff. The SDKs already do this. | | Permanent request error | `400`, `401`, `403`, `404`, `422` | Do not retry; fix the request or key. | | Low-confidence answer | HTTP 200, `confidence` below your gate | **Not an error.** Escalate (§5). | SDK defaults, both languages (raw/docs/sdk__python__api__retries.md, sdk__javascript__api__interfaces__RetryPolicy.md): 2 retries; backoff 0.5 s doubling to a 5 s cap, 0.25 jitter; statuses `{408, 429, *range(500, 600)}`; `Retry-After` / `retry-after-ms` honoured; connection errors and timeouts retried. Python adds a total budget `timeout = 30.0` s per call and a per-operation `DEFAULT_TIMEOUT = 10.0` s. Full table: [[reference/rate-limits-and-errors]]. The upstream override example `http_statuses={429, 500, 502, 503, 504}` **drops `529`**; keep the default range. mizchi (jev-test-filter) saw a real `529` (`system_overloaded`) outlast client retries; frequency unknown. **Idempotency.** No idempotency key exists in the docs or SDKs; retries carry an `X-TypeSafe-Retry-Count` header (raw/github/typesafe-sdk-python/src/typesafe_sdk/_core/constants.py, raw/github/typesafe-sdk-js/src/client.ts). Re-sending has no documented server-side effect; whether retries are billed is not stated. Dedupe decisions on (item ID, question version); side effects need their own record (*Scope*). **Never re-roll a low-confidence answer.** Repeats vary: mean per-answer probability SD `0.0098` (Choice) and `0.0102` (Noul) over 15 repeats; one Noul spanned `0.43` to `0.53`, crossing `0.5` (raw/docs/cookbooks__consistency_choice_cookbook.md, raw/docs/cookbooks__consistency_noul_cookbook.md). Re-asking until the gate clears biases the decision (inferred). Docs: "Escalate uncertain cases to a person or a more expensive reasoning model" (raw/docs/concepts__how-to-build-with-system-one.md). ## 4. Rate limits and batching | Limit (Jev 1.13) | Value | |---|---| | Throughput | 250,000 tokens per second | | Requests | 1,200 requests per minute | | Context | 64k tokens per request; 32k for `state` plus the longest question | | Choice options | at most 255 | | Score levels | API accepts up to 10 | Sources: raw/docs/models.md, api.md. Either limit returns `429`; limits "can change without notice"; higher limits via sales@typesafe.ai. Exceeding Usage Limits breaches MCA §2.3(j), grounds for suspension (§6; [[reference/legal-and-data]]): `429` is not free back-pressure. - **Batch questions against one state** ([[patterns/fan-out]]). In [[cookbooks/parallel-questions]], 13 separate calls changed no answer but re-sent the article 13 times: "the 13x token cost stays". - **Start concurrency small.** Cookbooks use pools of 4–16; two warn that rate limits start around eight concurrent calls (raw/docs/cookbooks__entity_alignment.md, cookbooks__autoresearch_feature_discovery.md). Nate B. Jones reports 25 concurrent calls with no `429`s via OpenRouter ([[ideas/measurements]]; community, one run). - **Keep offline work off the live path:** backfills and calibration runs get their own small pool (inferred). Per-key or per-account limits: not stated. Gateways add their own: kierandotai reports a small, slowly refilling Vercel AI Gateway free-tier quota (`unverified`). ## 5. Confidence gates and escalation Definitions (raw/docs/confidence.md, api.md): Choice and Score answers carry `probabilities` and `confidence`, a 0–1 statistic "derived from the probabilities". Noul answers have **no** `confidence`; `noul` is P(yes), so gate it with two cuts. Three paths: high, act; medium, confirm or review; low, "Do not act". Thresholds "scale with risk". Every docs threshold is illustrative: | Source | Threshold | |---|---| | raw/docs/confidence.md | floor `0.5`; `> 0.9` for a transfer | | [[patterns/confidence-routing]] | floor `0.6`; `> 0.85` for a transfer | | raw/docs/concepts__how-to-build-with-system-one.md | `0.8` in one step; `0.75` plus a spam band of `0.4 < risk < 0.6` in the worked example | | [[cookbooks/consistency-choice]] | top probability below `0.60` = `uncertain` | | [[cookbooks/consistency-noul]] | `0.30`–`0.70` = `uncertain` | | [[cookbooks/sde-cascade]] | escalate if any per-field P(wrong) is above `0.7` | Set yours empirically: "Test thresholds by plotting confidence against accuracy on your data" (raw/docs/concepts__how-to-build-with-system-one.md). Cascade ([[cookbooks/sde-cascade]]): take the `max` per-field P(wrong), so one red flag is not "averaged into silence". Track the escalation target's load (§7). ## 6. Replays and audits One record per decision: item ID; question version and spec hash; requested and returned `model` (raw/docs/models.md); `request_id` (`x-typesafe-request-id`); full `probabilities` / `confidence` / `noul`; threshold version and action; cache hit. Log `usage.input_tokens` and latency per parsed response: a cache hit has neither. **Replay from the log, never by re-asking:** re-run decision code over stored probabilities, as the cookbooks do with a shipped `json_cache.json` that "spends no API calls" ([[cookbooks/consistency-choice]]). Re-asking is not a replay: - No seed or temperature parameter; no source claims bit-identical repeats ([[syntheses/faq]] Q29). The docs promise "quantitatively similar outputs for semantically similar inputs" (raw/docs/model-jaggedness__jev-1.13.md). - Behind an alias, a later re-ask can reach a different model. ## 7. Monitoring | Signal | How | Note | |---|---|---| | Latency p50/p95 | Wall clock; Python SDK at `TYPESAFE_LOG_LEVEL=info` logs status, ms, request ID per request and each retry (raw/github/typesafe-sdk-python/src/typesafe_sdk/_core/transport.py) | Docs: "Most queries complete in about 100 ms"; cookbook means 111 and 114 ms. Community single-client p95s, none under load: 336–549 ms (wakegate, jev-papers, tocsin, Bouncer), 810 ms (chorylee), 444–1,006 ms over five days (jev-harness-router). Bouncer: ~193 ms of a 437 ms median is TCP+TLS setup: reuse one client (inferred). | | Error rate by class | `429`, `5xx`/`529`, connection, timeout, `422` separately; retries | A production `422` is a deploy bug: a code change broke a question. | | Model identity | Alert when returned `model` ≠ pin; never act on it (Implementation) | Never fires when pinned direct (inferred). Via OpenRouter, `~typesafe/jev-latest` came back as `typesafe/jev-1.13-20260917` (jev-papers, `unverified`): map gateway IDs explicitly. | | Answer-distribution drift | Per question version: label shares, mean `noul`, score histogram, mean `confidence` vs the calibration baseline | Catches input drift, state-builder bugs, model changes (inferred). | | Escalation-rate drift | Share below the gate, per pipeline | Rising: inputs moved or a question degraded. Falling: maybe a bug bypassing the gate (inferred). | | Estimated cost per pipeline per day | Σ reported `usage.input_tokens` × $0.042 / 1,000,000 over parsed responses (not cache hits); count responses with no reported usage separately | Output tokens are free (raw/docs/models.md). Failed attempts: not logged, billing undocumented. Without auto-refill, TypeSafe "may decline to generate Output" when Credits run out (MCA §8.2). | | Human spot-check | Weekly labelled sample of auto-acted decisions, added to the calibration set | Community variant: [Nate B. Jones](https://unlock-ai.natebjones.com/guides/jev-shaped-problems)'s shadow mode ([[guides/testing-and-evaluation]]). | No doc gives alert thresholds; derive them from your baseline (community example: jeval's `drift --fail-on ece-increase=0.05`). ## 8. Fallbacks and model upgrades No uptime SLA; the Services are "AS IS" and "AS AVAILABLE" (MCA §9.3; [[reference/legal-and-data]]). | Pipeline kind | On `retry_later` (unreachable after retries, or wrong model) | |---|---| | Offline or batch | Queue the item, retry later, alert if the queue ages. | | Live, low stakes | Degrade: take the safe default or the old deterministic path. | | Live, high stakes or irreversible | Route to a human. Never auto-act without an answer. | | Guardrail or verifier | Fail closed: treat "no answer" as "escalate" (inferred). | [[reference/system-one-adapter]] (an OpenAI, Anthropic or Gemini LLM behind the `typesafe_sdk` interface) is documented for comparison, not failover; Jev-calibrated thresholds do not transfer (inferred). Gateway base URLs (OpenRouter, Vercel AI Gateway; raw/docs/sdk__python__usage.md) are not stated to be independent of TypeSafe's availability. **Model upgrade procedure:** 1. Stay pinned. Watch the Models page and `jev-preview`, which "moves ahead of `jev-latest` when a preview build is available"; in the docs fetched 2026-09-22 both point to `jev-1.13.0` (raw/docs/models.md). 2. Run your calibration set on the new ID in shadow beside the pinned one. 3. Rebuild confidence buckets, re-pick thresholds, compare answer distributions. 4. Bump the question/threshold version with the pin; old cache entries fall out through the key. 5. Flip, keep the old ID for rollback, watch §7 (inferred). Diogo Almeida (first-party): "We will not change our models when we deploy them", but "not promising long-term support" ([[entities/latent-space-interview]]). The docs state no deprecation policy for versioned IDs; MCA §2.5 promises "commercially reasonable efforts" at advance notice of materially adverse API updates. ## 9. Data handling What leaves your systems: `state`, `questions` and answers (MCA Input and Output, together Customer Data; [[reference/legal-and-data]]). - **Training.** "Jev is not trained on customer requests or responses" (raw/docs/models.md); MCA §4.1 adds "without Customer's prior consent". - **Telemetry** (logs, hashes, summary statistics, classifications) may be processed "without restriction" (MCA §4.3), perpetually since the 2026-09-19 MCA. - **Retention.** No period is stated; TypeSafe "may delete Customer Data at any time" (MCA §10.3). Zero data retention is offered "for enterprise customers" via privacy@typesafe.ai (raw/docs/legal.md). Vercel's AI Gateway guide lists ZDR and No Training "per request" for `typesafe-ai/jev` (third-party, `unverified`). - **Hosting:** United States. The DPA lists sensitive data as N/A. Minimise `state` (it helps accuracy too: [[concepts/jaggedness-jev-1-13]]). `TYPESAFE_LOG_LEVEL=debug` logs request and response bodies (0.7.1 redacts credentials, not payloads; [[reference/python-sdk-internals]]): run `info` or higher. Your retention policy covers cache and audit log; a gateway adds a third party to the data path (inferred). ## Implementation: a versioned, cached, logged, gated decision Python, `typesafe-sdk` 0.7.1, `TYPESAFE_API_KEY` set, standard-library SQLite. Decides, gates and logs; delivery is out of scope (*Scope*). ```python import hashlib import json import logging import sqlite3 import time from typesafe_sdk import ( Choice, RetryPolicy, TypeSafeAPIConnectionError, TypeSafeAPIError, TypeSafeClient, TypeSafeError, ) log = logging.getLogger("jev_ops") # --- 1. Everything that shapes the decision, versioned together ------------- MODEL = "jev-1.13.0" # pinned versioned ID, not the jev-latest alias ACCEPTED_MODELS = {MODEL} # via a gateway, add its returned ID here explicitly (section 7) DECISION_VERSION = "ticket-routing@3" # bump on ANY change below QUESTIONS = { "department": Choice( instructions="Which team should handle this?", criteria={ # option order is part of the version "billing": "Payments, invoicing, refunds", "technical": "Bugs, outages, integrations", "sales": "Pricing, upgrades, new accounts", }, ), } MIN_CONFIDENCE = 0.8 # picked from YOUR calibration buckets for this version TRANSIENT = {408, 429} | set(range(500, 600)) # same set the SDK retries USD_PER_INPUT_TOKEN = 0.042 / 1_000_000 # output tokens are free SPEC = json.dumps( {"model": MODEL, "questions": {k: q.model_dump(mode="json") for k, q in QUESTIONS.items()}} ) # no sort_keys: option order must stay in the key SPEC_HASH = hashlib.sha256(SPEC.encode()).hexdigest()[:16] # One file per decision version: a new schema never meets an old file (no migration). db = sqlite3.connect(f"jev_ops_{DECISION_VERSION}.db") db.executescript( """ CREATE TABLE IF NOT EXISTS cache (key TEXT PRIMARY KEY, response TEXT); -- one row per PARSED response, not per attempt: estimated spend, cache hits excluded; -- input_tokens is NULL when the API reported no usage CREATE TABLE IF NOT EXISTS calls ( ts REAL, decision_version TEXT, request_id TEXT, answered_model TEXT, input_tokens INTEGER, latency_ms REAL ); CREATE TABLE IF NOT EXISTS decisions ( item_id TEXT, decision_version TEXT, spec_hash TEXT, requested_model TEXT, answered_model TEXT, request_id TEXT, answers TEXT, threshold REAL, action TEXT, cache_hit INTEGER, ts REAL, PRIMARY KEY (item_id, decision_version) -- dedupes the DECISION, not delivery ); -- illustrative log of retryable failures; nothing in this sample consumes it CREATE TABLE IF NOT EXISTS retry_queue ( item_id TEXT, decision_version TEXT, state TEXT, last_reason TEXT, attempts INTEGER, ts REAL, PRIMARY KEY (item_id, decision_version) ); """ ) client = TypeSafeClient(model=MODEL, retry=RetryPolicy(max_retries=2, timeout=30.0)) def evaluate(state: dict) -> tuple[dict, bool, str | None]: """Return (response_dict, cache_hit, request_id).""" key = hashlib.sha256((SPEC_HASH + json.dumps(state)).encode()).hexdigest() row = db.execute("SELECT response FROM cache WHERE key = ?", (key,)).fetchone() if row: return json.loads(row[0]), True, None # no API call, nothing billed started = time.monotonic() response = client.system_one(state, QUESTIONS) latency_ms = (time.monotonic() - started) * 1000 payload = response.model_dump(mode="json") # model, usage, answers try: request_id = response.request_id except TypeSafeError: request_id = None db.execute( # reported usage, whichever model answered (input_tokens may be None) "INSERT INTO calls VALUES (?,?,?,?,?,?)", (time.time(), DECISION_VERSION, request_id, payload["model"], payload["usage"]["input_tokens"], latency_ms), ) if payload["model"] in ACCEPTED_MODELS: # never cache an answer from another model db.execute("INSERT OR REPLACE INTO cache VALUES (?, ?)", (key, json.dumps(payload))) db.commit() return payload, False, request_id def route_ticket(item_id: str, state: dict) -> str: """Return "route:" or "human_review" (both final, stored) or "retry_later" (nothing stored; call again). item_id is treated as immutable: a later call with a changed state for the same item_id returns the stored action.""" done = db.execute( "SELECT action FROM decisions WHERE item_id = ? AND decision_version = ?", (item_id, DECISION_VERSION), ).fetchone() if done: # already decided (routed or human_review): return it, do not decide twice return done[0] try: payload, cache_hit, request_id = evaluate(state) except TypeSafeAPIConnectionError: # network down after SDK retries return retry_later(item_id, state, "unreachable") except TypeSafeAPIError as error: if error.status in TRANSIENT: # still failing after SDK retries (e.g. 529) return retry_later(item_id, state, f"http_{error.status}") raise # 400/401/403/404/422: a bug or a key problem, not an outage # --- Model gate: before the cache is trusted and before any threshold --- if payload["model"] not in ACCEPTED_MODELS: log.warning("item %s answered by %r, pinned %r", item_id, payload["model"], MODEL) return retry_later(item_id, state, f"model:{payload['model']}") answer = payload["answers"]["department"] # --- 5. Confidence gate: escalate, never re-ask --- if answer["confidence"] < MIN_CONFIDENCE: action = "human_review" # final: stored below, so later calls return it else: action = f"route:{answer['choice']}" # --- 6. Decision record, committed before the caller delivers anything --- db.execute( "INSERT INTO decisions VALUES (?,?,?,?,?,?,?,?,?,?,?)", ( item_id, DECISION_VERSION, SPEC_HASH, MODEL, payload["model"], request_id, json.dumps(payload["answers"]), MIN_CONFIDENCE, action, int(cache_hit), time.time(), ), ) db.commit() return action # the caller delivers it; this sample does not (see Scope) def retry_later(item_id: str, state: dict, reason: str) -> str: """No decision row is written, so the next route_ticket call asks again.""" db.execute( """INSERT INTO retry_queue VALUES (?, ?, ?, ?, 1, ?) ON CONFLICT (item_id, decision_version) DO UPDATE SET state = excluded.state, last_reason = excluded.last_reason, attempts = attempts + 1, ts = excluded.ts""", (item_id, DECISION_VERSION, json.dumps(state), reason, time.time()), ) db.commit() return "retry_later" # not final: the caller decides what to do meanwhile def replay(new_threshold: float) -> dict: """Re-decide logged answers under a new threshold. No API calls.""" counts: dict[str, int] = {} for (answers,) in db.execute( "SELECT answers FROM decisions WHERE decision_version = ?", (DECISION_VERSION,) ): a = json.loads(answers)["department"] action = "human_review" if a["confidence"] < new_threshold else f"route:{a['choice']}" counts[action] = counts.get(action, 0) + 1 return counts def cost_usd(since_ts: float) -> tuple[float, int]: """(estimated USD from reported usage on parsed responses, responses with unknown usage). Cache hits are excluded; failed attempts are not recorded. Unknown usage is not zero.""" tokens, unknown = db.execute( "SELECT COALESCE(SUM(input_tokens), 0), COUNT(*) - COUNT(input_tokens)" " FROM calls WHERE ts >= ?", (since_ts,), ).fetchone() return tokens * USD_PER_INPUT_TOKEN, unknown if __name__ == "__main__": ticket = {"message": "Help! My payouts have been failing for 3 days."} print(route_ticket("ticket-0001", ticket)) print(replay(new_threshold=0.9)) print(cost_usd(since_ts=time.time() - 86_400)) # (usd, unknown_usage_count) client.close() ``` **Final vs retryable.** `route:` and low-confidence `human_review` are final: stored, returned by later calls without an API call. Transport failure after SDK retries, or an answer from a model outside `ACCEPTED_MODELS`, returns `retry_later`: nothing decided, cached or thresholded; the next call asks again. The return value tells the caller which; it picks the interim action (§8). `retry_queue` is an illustrative log (keyed per item and version, `attempts` counted); the sample has no consumer, and a real system needs one with identity and lifecycle (inferred). **Cost** is estimated spend from the usage the API reported on parsed responses (gate-rejected included); cache hits excluded; failed attempts are not recorded, and whether they are billed is undocumented. `input_tokens` is `int | None` ([[reference/python-sdk-responses]]): `cost_usd` also counts unknown-usage responses, so an unreported window never reads as zero. **Database.** One file per `DECISION_VERSION`. The sample does not migrate old databases: a changed schema or question version gets a new file; old rows are kept in the old one, never deleted. **Scope: decision, not delivery.** The `(item_id, decision_version)` key dedupes the *decision*, not the side effect: a crash after the commit, or a caller retry, can miss or repeat delivery, which needs its own pending/delivered record keyed for the receiver (inferred). `item_id` is immutable: a changed `state` is ignored; if items change, version the `item_id`. **How checked:** 24/24 checks, `typesafe-sdk` 0.7.1, mocked HTTP server, 2026-09-23: every path above, plus an old 13-column `jev_ops.db` left untouched, a repeated `529` (attempts upserted, later success decides) and `None` usage. Not run live. Script: `docs/verification/production-operations-sample-test.py` (repository only). ## Related - [[guides/testing-and-evaluation]]: calibration set - [[reference/rate-limits-and-errors]], [[reference/python-sdk-retries-errors]], [[reference/javascript-sdk-errors]]: statuses, retries, exceptions - [[reference/models-and-pricing]]: IDs, aliases, prices - [[concepts/confidence]], [[patterns/confidence-routing]], [[cookbooks/sde-cascade]]: gates, escalation - [[concepts/jaggedness-jev-1-13]]: why wording is versioned - [[reference/legal-and-data]]: MCA, DPA, ZDR - [[guides/agent-integration-playbook]], [[syntheses/version-timeline]]: build procedure, versions ## Sources - raw/docs/api.md, models.md, confidence.md, concepts__how-to-build-with-system-one.md, model-jaggedness__jev-1.13.md, patterns__confidence-routing.md, legal.md (https://docs.typesafe.ai/) - raw/docs/cookbooks__consistency_choice_cookbook.md, cookbooks__consistency_noul_cookbook.md, cookbooks__sde_cascade.md, cookbooks__parallel_questions.md, cookbooks__entity_alignment.md, cookbooks__autoresearch_feature_discovery.md, cookbooks__date_extraction_cookbook.md (https://docs.typesafe.ai/cookbooks) - raw/docs/sdk__python__api__retries.md, sdk__python__api__exceptions.md, sdk__python__api__constants.md, sdk__python__usage.md, sdk__javascript__api__interfaces__RetryPolicy.md - raw/github/typesafe-sdk-python/src/typesafe_sdk/_core/transport.py and constants.py; raw/github/typesafe-sdk-js/src/client.ts - raw/site/typesafe-ai-legal_mca.txt (https://typesafe.ai/legal/mca) - raw/community/latent-space-jev-diogo-almeida.md (https://www.latent.space/p/jev): first-party, not documentation - raw/community/archerhume-jevs-architecture-unmasked.md (https://archerhume.com/posts/jevs-architecture-unmasked): community - Community figures via [[ideas/measurements]] (Nate B. Jones) - 2026-09-23 sweep (community tier): the raw/community/ and raw/x-repos/ files in frontmatter `sources:`