Running Jev in production: versioning, caching, retries, monitoring and fallbacks

TL;DR Pin model to a versioned ID (jev-1.13.0); version questions, option lists and thresholds together. Cache on exact state + question spec + model ID. Let the SDK retry transport failures (408, 429, 5xx, connection, timeout). Act only on answers from the pinned model; escalate low-confidence answers, never re-ask. Log answers with question version, returned model, probabilities, request ID; replay from the log. Monitor latency, errors, drift, cost. Plan fallbacks: no uptime SLA.

Unsourced advice is marked "(inferred)". Labelled calibration set: Testing and evaluating a Jev workflow.

1. Version everything that shapes a decision

One version string per decision covers:

• instructions text, including structured-instruction fields.
• criteria: Choice option keys, their descriptions and their order; Score levels and their order; Noul true/false descriptions.
• Question type: "Don't carry a threshold tuned on a Noul over to a Choice" (raw/docs/model-jaggedness__jev-1.13.md).
• Thresholds and composition weights in your code.
• The state builder: which fields, in what order, serialised how.
• model: a versioned ID, not an alias.

Why:

• A reworded question is a new instrument. The same question as a Noul and as a yes/no Choice gave noul 0.22 against yes 0.01; a negated Noul pair summed to 1.19 (raw/docs/model-jaggedness__jev-1.13.md). Thresholds tuned on version N do not carry to N+1 (inferred).
• Options interact. Archer Hume: adding an option shifted the odds between existing ones; reversing option order moved a classification from ~0.84–0.89 to 0.93–0.96 (community, unverified).
• State order matters too (docs silent). TJ Klug: a Noul read 0.33 with goal after a kilobyte of diff, 0.93 with goal first. Reversing 30 passages in state changed Jev Choice's top pick on 24.7% of 1,617 queries (anessbelbati/jev-rerank-bench). Community, unverified.
• Aliases move. "An alias moves when a new release ships"; with tuned thresholds, "pin that version's ID" (raw/docs/models.md). jev-latest is the SDK default (raw/docs/sdk__python__api__constants.md).
• Question IDs are free: "not sent to the underlying model" (raw/docs/api.md).

GET /v1/models "currently lists the aliases", not what they resolve to; read that from each response's model.

2. Caching

Key on the full request:

1. The exact serialised state.
2. The full question spec (type, instructions, criteria) in the order you send it.
3. The pinned model ID.

Do not sort keys in criteria or state: order is part of the instrument (§1); a cache that canonicalises key order (JevGuard does) can merge requests Jev answers differently (inferred). The cookbooks key on a digest of "everything that shapes the prompt/rubric" and keep the returned model "because an alias can resolve to a different version later" (raw/docs/cookbooks__consistency_noul_cookbook.md).

Do not cache: errors and retry_later outcomes; a response whose model differs from your pin; requests carrying a nonce (the consistency cookbooks add a uid to force fresh draws); calibration runs; answers depending on a fact not in state, such as "now" (put the date in state, as Cookbook: Date extraction does).

TTLs. None in the docs. Community: jev-axi reuses answers up to 24 h while jev-latest resolves to the same version; tocsin replaces verdicts when a newer model answers. Pinned, an exact-input entry is valid until the question version or pin changes; behind an alias it can silently belong to an older model: pin rather than shorten the TTL (inferred).

3. Retries and errors

SDK defaults, both languages (raw/docs/sdk__python__api__retries.md, sdk__javascript__api__interfaces__RetryPolicy.md): 2 retries; backoff 0.5 s doubling to a 5 s cap, 0.25 jitter; statuses {408, 429, *range(500, 600)}; Retry-After / retry-after-ms honoured; connection errors and timeouts retried. Python adds a total budget timeout = 30.0 s per call and a per-operation DEFAULT_TIMEOUT = 10.0 s. Full table: HTTP status codes, rate limits, retry semantics. The upstream override example http_statuses={429, 500, 502, 503, 504} drops 529; keep the default range. mizchi (jev-test-filter) saw a real 529 (system_overloaded) outlast client retries; frequency unknown.

Idempotency. No idempotency key exists in the docs or SDKs; retries carry an X-TypeSafe-Retry-Count header (raw/github/typesafe-sdk-python/src/typesafe_sdk/_core/constants.py, raw/github/typesafe-sdk-js/src/client.ts). Re-sending has no documented server-side effect; whether retries are billed is not stated. Dedupe decisions on (item ID, question version); side effects need their own record (Scope).

Never re-roll a low-confidence answer. Repeats vary: mean per-answer probability SD 0.0098 (Choice) and 0.0102 (Noul) over 15 repeats; one Noul spanned 0.43 to 0.53, crossing 0.5 (raw/docs/cookbooks__consistency_choice_cookbook.md, raw/docs/cookbooks__consistency_noul_cookbook.md). Re-asking until the gate clears biases the decision (inferred). Docs: "Escalate uncertain cases to a person or a more expensive reasoning model" (raw/docs/concepts__how-to-build-with-system-one.md).

4. Rate limits and batching

Sources: raw/docs/models.md, api.md. Either limit returns 429; limits "can change without notice"; higher limits via sales@typesafe.ai. Exceeding Usage Limits breaches MCA §2.3(j), grounds for suspension (§6; Legal: MCA, DPA, privacy, data retention): 429 is not free back-pressure.

• Batch questions against one state (Speculative fan-out). In Cookbook: Parallel questions, 13 separate calls changed no answer but re-sent the article 13 times: "the 13x token cost stays".
• Start concurrency small. Cookbooks use pools of 4–16; two warn that rate limits start around eight concurrent calls (raw/docs/cookbooks__entity_alignment.md, cookbooks__autoresearch_feature_discovery.md). Nate B. Jones reports 25 concurrent calls with no 429s via OpenRouter (Measurements, access routes and open replicas; community, one run).
• Keep offline work off the live path: backfills and calibration runs get their own small pool (inferred). Per-key or per-account limits: not stated. Gateways add their own: kierandotai reports a small, slowly refilling Vercel AI Gateway free-tier quota (unverified).

5. Confidence gates and escalation

Definitions (raw/docs/confidence.md, api.md): Choice and Score answers carry probabilities and confidence, a 0–1 statistic "derived from the probabilities". Noul answers have no confidence; noul is P(yes), so gate it with two cuts. Three paths: high, act; medium, confirm or review; low, "Do not act". Thresholds "scale with risk". Every docs threshold is illustrative:

Set yours empirically: "Test thresholds by plotting confidence against accuracy on your data" (raw/docs/concepts__how-to-build-with-system-one.md).

Cascade (Cookbook: SDE cascade): take the max per-field P(wrong), so one red flag is not "averaged into silence". Track the escalation target's load (§7).

6. Replays and audits

One record per decision: item ID; question version and spec hash; requested and returned model (raw/docs/models.md); request_id (x-typesafe-request-id); full probabilities / confidence / noul; threshold version and action; cache hit. Log usage.input_tokens and latency per parsed response: a cache hit has neither.

Replay from the log, never by re-asking: re-run decision code over stored probabilities, as the cookbooks do with a shipped json_cache.json that "spends no API calls" (Cookbook: Self-consistency — choices). Re-asking is not a replay:

• No seed or temperature parameter; no source claims bit-identical repeats (FAQ for agents and developers Q29). The docs promise "quantitatively similar outputs for semantically similar inputs" (raw/docs/model-jaggedness__jev-1.13.md).
• Behind an alias, a later re-ask can reach a different model.

7. Monitoring

No doc gives alert thresholds; derive them from your baseline (community example: jeval's drift --fail-on ece-increase=0.05).

8. Fallbacks and model upgrades

No uptime SLA; the Services are "AS IS" and "AS AVAILABLE" (MCA §9.3; Legal: MCA, DPA, privacy, data retention).

system-one-adapter: LLM-backed drop-in for TypeSafeClient (an OpenAI, Anthropic or Gemini LLM behind the typesafe_sdk interface) is documented for comparison, not failover; Jev-calibrated thresholds do not transfer (inferred). Gateway base URLs (OpenRouter, Vercel AI Gateway; raw/docs/sdk__python__usage.md) are not stated to be independent of TypeSafe's availability.

Model upgrade procedure:

1. Stay pinned. Watch the Models page and jev-preview, which "moves ahead of jev-latest when a preview build is available"; in the docs fetched 2026-09-22 both point to jev-1.13.0 (raw/docs/models.md).
2. Run your calibration set on the new ID in shadow beside the pinned one.
3. Rebuild confidence buckets, re-pick thresholds, compare answer distributions.
4. Bump the question/threshold version with the pin; old cache entries fall out through the key.
5. Flip, keep the old ID for rollback, watch §7 (inferred).

Diogo Almeida (first-party): "We will not change our models when we deploy them", but "not promising long-term support" (Latent Space interview with Diogo Almeida (2026-09-21)). The docs state no deprecation policy for versioned IDs; MCA §2.5 promises "commercially reasonable efforts" at advance notice of materially adverse API updates.

9. Data handling

What leaves your systems: state, questions and answers (MCA Input and Output, together Customer Data; Legal: MCA, DPA, privacy, data retention).

• Training. "Jev is not trained on customer requests or responses" (raw/docs/models.md); MCA §4.1 adds "without Customer's prior consent".
• Telemetry (logs, hashes, summary statistics, classifications) may be processed "without restriction" (MCA §4.3), perpetually since the 2026-09-19 MCA.
• Retention. No period is stated; TypeSafe "may delete Customer Data at any time" (MCA §10.3). Zero data retention is offered "for enterprise customers" via privacy@typesafe.ai (raw/docs/legal.md). Vercel's AI Gateway guide lists ZDR and No Training "per request" for typesafe-ai/jev (third-party, unverified).
• Hosting: United States. The DPA lists sensitive data as N/A.

Minimise state (it helps accuracy too: Jev 1.13 jaggedness: known failure modes). TYPESAFE_LOG_LEVEL=debug logs request and response bodies (0.7.1 redacts credentials, not payloads; Python SDK internals: headers, key validation, gateways and base URLs, logging, forward compatibility, dependencies, migration notes): run info or higher. Your retention policy covers cache and audit log; a gateway adds a third party to the data path (inferred).

Implementation: a versioned, cached, logged, gated decision

Python, typesafe-sdk 0.7.1, TYPESAFE_API_KEY set, standard-library SQLite. Decides, gates and logs; delivery is out of scope (Scope).

import hashlib
import json
import logging
import sqlite3
import time

from typesafe_sdk import (
    Choice,
    RetryPolicy,
    TypeSafeAPIConnectionError,
    TypeSafeAPIError,
    TypeSafeClient,
    TypeSafeError,
)

log = logging.getLogger("jev_ops")

# --- 1. Everything that shapes the decision, versioned together -------------
MODEL = "jev-1.13.0"  # pinned versioned ID, not the jev-latest alias
ACCEPTED_MODELS = {MODEL}  # via a gateway, add its returned ID here explicitly (section 7)
DECISION_VERSION = "ticket-routing@3"  # bump on ANY change below
QUESTIONS = {
    "department": Choice(
        instructions="Which team should handle this?",
        criteria={  # option order is part of the version
            "billing": "Payments, invoicing, refunds",
            "technical": "Bugs, outages, integrations",
            "sales": "Pricing, upgrades, new accounts",
        },
    ),
}
MIN_CONFIDENCE = 0.8  # picked from YOUR calibration buckets for this version
TRANSIENT = {408, 429} | set(range(500, 600))  # same set the SDK retries
USD_PER_INPUT_TOKEN = 0.042 / 1_000_000  # output tokens are free

SPEC = json.dumps(
    {"model": MODEL, "questions": {k: q.model_dump(mode="json") for k, q in QUESTIONS.items()}}
)  # no sort_keys: option order must stay in the key
SPEC_HASH = hashlib.sha256(SPEC.encode()).hexdigest()[:16]

# One file per decision version: a new schema never meets an old file (no migration).
db = sqlite3.connect(f"jev_ops_{DECISION_VERSION}.db")
db.executescript(
    """
    CREATE TABLE IF NOT EXISTS cache (key TEXT PRIMARY KEY, response TEXT);
    -- one row per PARSED response, not per attempt: estimated spend, cache hits excluded;
    -- input_tokens is NULL when the API reported no usage
    CREATE TABLE IF NOT EXISTS calls (
        ts REAL, decision_version TEXT, request_id TEXT, answered_model TEXT,
        input_tokens INTEGER, latency_ms REAL
    );
    CREATE TABLE IF NOT EXISTS decisions (
        item_id TEXT, decision_version TEXT, spec_hash TEXT,
        requested_model TEXT, answered_model TEXT, request_id TEXT,
        answers TEXT, threshold REAL, action TEXT, cache_hit INTEGER, ts REAL,
        PRIMARY KEY (item_id, decision_version)  -- dedupes the DECISION, not delivery
    );
    -- illustrative log of retryable failures; nothing in this sample consumes it
    CREATE TABLE IF NOT EXISTS retry_queue (
        item_id TEXT, decision_version TEXT, state TEXT, last_reason TEXT,
        attempts INTEGER, ts REAL,
        PRIMARY KEY (item_id, decision_version)
    );
    """
)

client = TypeSafeClient(model=MODEL, retry=RetryPolicy(max_retries=2, timeout=30.0))


def evaluate(state: dict) -> tuple[dict, bool, str | None]:
    """Return (response_dict, cache_hit, request_id)."""
    key = hashlib.sha256((SPEC_HASH + json.dumps(state)).encode()).hexdigest()
    row = db.execute("SELECT response FROM cache WHERE key = ?", (key,)).fetchone()
    if row:
        return json.loads(row[0]), True, None  # no API call, nothing billed
    started = time.monotonic()
    response = client.system_one(state, QUESTIONS)
    latency_ms = (time.monotonic() - started) * 1000
    payload = response.model_dump(mode="json")  # model, usage, answers
    try:
        request_id = response.request_id
    except TypeSafeError:
        request_id = None
    db.execute(  # reported usage, whichever model answered (input_tokens may be None)
        "INSERT INTO calls VALUES (?,?,?,?,?,?)",
        (time.time(), DECISION_VERSION, request_id, payload["model"],
         payload["usage"]["input_tokens"], latency_ms),
    )
    if payload["model"] in ACCEPTED_MODELS:  # never cache an answer from another model
        db.execute("INSERT OR REPLACE INTO cache VALUES (?, ?)", (key, json.dumps(payload)))
    db.commit()
    return payload, False, request_id


def route_ticket(item_id: str, state: dict) -> str:
    """Return "route:<team>" or "human_review" (both final, stored) or "retry_later"
    (nothing stored; call again). item_id is treated as immutable: a later call with a
    changed state for the same item_id returns the stored action."""
    done = db.execute(
        "SELECT action FROM decisions WHERE item_id = ? AND decision_version = ?",
        (item_id, DECISION_VERSION),
    ).fetchone()
    if done:  # already decided (routed or human_review): return it, do not decide twice
        return done[0]
    try:
        payload, cache_hit, request_id = evaluate(state)
    except TypeSafeAPIConnectionError:  # network down after SDK retries
        return retry_later(item_id, state, "unreachable")
    except TypeSafeAPIError as error:
        if error.status in TRANSIENT:  # still failing after SDK retries (e.g. 529)
            return retry_later(item_id, state, f"http_{error.status}")
        raise  # 400/401/403/404/422: a bug or a key problem, not an outage

    # --- Model gate: before the cache is trusted and before any threshold ---
    if payload["model"] not in ACCEPTED_MODELS:
        log.warning("item %s answered by %r, pinned %r", item_id, payload["model"], MODEL)
        return retry_later(item_id, state, f"model:{payload['model']}")

    answer = payload["answers"]["department"]
    # --- 5. Confidence gate: escalate, never re-ask ---
    if answer["confidence"] < MIN_CONFIDENCE:
        action = "human_review"  # final: stored below, so later calls return it
    else:
        action = f"route:{answer['choice']}"

    # --- 6. Decision record, committed before the caller delivers anything ---
    db.execute(
        "INSERT INTO decisions VALUES (?,?,?,?,?,?,?,?,?,?,?)",
        (
            item_id, DECISION_VERSION, SPEC_HASH, MODEL, payload["model"], request_id,
            json.dumps(payload["answers"]), MIN_CONFIDENCE, action, int(cache_hit), time.time(),
        ),
    )
    db.commit()
    return action  # the caller delivers it; this sample does not (see Scope)


def retry_later(item_id: str, state: dict, reason: str) -> str:
    """No decision row is written, so the next route_ticket call asks again."""
    db.execute(
        """INSERT INTO retry_queue VALUES (?, ?, ?, ?, 1, ?)
           ON CONFLICT (item_id, decision_version) DO UPDATE SET state = excluded.state,
           last_reason = excluded.last_reason, attempts = attempts + 1, ts = excluded.ts""",
        (item_id, DECISION_VERSION, json.dumps(state), reason, time.time()),
    )
    db.commit()
    return "retry_later"  # not final: the caller decides what to do meanwhile


def replay(new_threshold: float) -> dict:
    """Re-decide logged answers under a new threshold. No API calls."""
    counts: dict[str, int] = {}
    for (answers,) in db.execute(
        "SELECT answers FROM decisions WHERE decision_version = ?", (DECISION_VERSION,)
    ):
        a = json.loads(answers)["department"]
        action = "human_review" if a["confidence"] < new_threshold else f"route:{a['choice']}"
        counts[action] = counts.get(action, 0) + 1
    return counts


def cost_usd(since_ts: float) -> tuple[float, int]:
    """(estimated USD from reported usage on parsed responses, responses with unknown usage).
    Cache hits are excluded; failed attempts are not recorded. Unknown usage is not zero."""
    tokens, unknown = db.execute(
        "SELECT COALESCE(SUM(input_tokens), 0), COUNT(*) - COUNT(input_tokens)"
        " FROM calls WHERE ts >= ?",
        (since_ts,),
    ).fetchone()
    return tokens * USD_PER_INPUT_TOKEN, unknown


if __name__ == "__main__":
    ticket = {"message": "Help! My payouts have been failing for 3 days."}
    print(route_ticket("ticket-0001", ticket))
    print(replay(new_threshold=0.9))
    print(cost_usd(since_ts=time.time() - 86_400))  # (usd, unknown_usage_count)
    client.close()

Final vs retryable. route:<team> and low-confidence human_review are final: stored, returned by later calls without an API call. Transport failure after SDK retries, or an answer from a model outside ACCEPTED_MODELS, returns retry_later: nothing decided, cached or thresholded; the next call asks again. The return value tells the caller which; it picks the interim action (§8). retry_queue is an illustrative log (keyed per item and version, attempts counted); the sample has no consumer, and a real system needs one with identity and lifecycle (inferred).

Cost is estimated spend from the usage the API reported on parsed responses (gate-rejected included); cache hits excluded; failed attempts are not recorded, and whether they are billed is undocumented. input_tokens is int | None (Python SDK responses, answers, usage, models): cost_usd also counts unknown-usage responses, so an unreported window never reads as zero.

Database. One file per DECISION_VERSION. The sample does not migrate old databases: a changed schema or question version gets a new file; old rows are kept in the old one, never deleted.

Scope: decision, not delivery. The (item_id, decision_version) key dedupes the decision, not the side effect: a crash after the commit, or a caller retry, can miss or repeat delivery, which needs its own pending/delivered record keyed for the receiver (inferred). item_id is immutable: a changed state is ignored; if items change, version the item_id.

How checked: 24/24 checks, typesafe-sdk 0.7.1, mocked HTTP server, 2026-09-23: every path above, plus an old 13-column jev_ops.db left untouched, a repeated 529 (attempts upserted, later success decides) and None usage. Not run live. Script: docs/verification/production-operations-sample-test.py (repository only).

Related

• Testing and evaluating a Jev workflow: calibration set
• HTTP status codes, rate limits, retry semantics, Python SDK retries, exceptions, constants, JavaScript SDK error classes, RetryPolicy, RequestOptions: statuses, retries, exceptions
• Models, aliases, pricing, rate limits, context: IDs, aliases, prices
• Confidence vs probability, Confidence-gated routing, Cookbook: SDE cascade: gates, escalation
• Jev 1.13 jaggedness: known failure modes: why wording is versioned
• Legal: MCA, DPA, privacy, data retention: MCA, DPA, ZDR
• Playbook for LLM agents building with Jev, Versions and timeline (models, SDKs, API, company): build procedure, versions

Sources

• raw/docs/api.md, models.md, confidence.md, concepts__how-to-build-with-system-one.md, model-jaggedness__jev-1.13.md, patterns__confidence-routing.md, legal.md (https://docs.typesafe.ai/)
• raw/docs/cookbooks__consistency_choice_cookbook.md, cookbooks__consistency_noul_cookbook.md, cookbooks__sde_cascade.md, cookbooks__parallel_questions.md, cookbooks__entity_alignment.md, cookbooks__autoresearch_feature_discovery.md, cookbooks__date_extraction_cookbook.md (https://docs.typesafe.ai/cookbooks)
• raw/docs/sdk__python__api__retries.md, sdk__python__api__exceptions.md, sdk__python__api__constants.md, sdk__python__usage.md, sdk__javascript__api__interfaces__RetryPolicy.md
• raw/github/typesafe-sdk-python/src/typesafe_sdk/_core/transport.py and constants.py; raw/github/typesafe-sdk-js/src/client.ts
• raw/site/typesafe-ai-legal_mca.txt (https://typesafe.ai/legal/mca)
• raw/community/latent-space-jev-diogo-almeida.md (https://www.latent.space/p/jev): first-party, not documentation
• raw/community/archerhume-jevs-architecture-unmasked.md (https://archerhume.com/posts/jevs-architecture-unmasked): community
• Community figures via Measurements, access routes and open replicas (Nate B. Jones)
• 2026-09-23 sweep (community tier): the raw/community/ and raw/x-repos/ files in frontmatter sources: