When not to use Jev: rules, embeddings, trained classifiers, small and frontier LLMs

TL;DR Jev for closed-set language judgements at volume that code consumes; code for anything computable exactly; an LLM for anything generated or reasoned over several steps; a person for uncertain or irreversible cases. Default design: rules floor → Jev → escalate on confidence. The sources barely discuss embeddings or trained classifiers as rivals; those claims are labelled (inferred) or community.

The question

"For this decision, is Jev the right tool, or does something else win — and when should they be combined?" Jev vs an LLM in JSON mode is covered in Jev vs LLM JSON mode / structured outputs; this page covers the wider field.

Labels: docs (wins every conflict), first-party (TypeSafe staff outside the docs), community (verified / contradicts docs / unverified). Every number is one attributed run.

What Jev says it is bad at

From Jev 1.13 jaggedness: known failure modes and System One Models (docs):

The design rule behind the table, from How to build software with System One: "Keep control flow, deterministic rules, and side effects in code." Step one there is "Use code when you can… It is reliable and cheap."

Alternative by alternative

Deterministic code, rules, regex

• Beats Jev: anything exact, such as arithmetic, dates, counting, identities, schema checks and side effects (docs, above). Jev does not guarantee identities: P(refund) + P(not refund) summed to 1.19 in the docs' own example (Jev 1.13 jaggedness: known failure modes §8).
• Jev beats it: language judgement that keywords miss. In Nate B. Jones's tests, keyword rules got 4/12 and Jev got 11/12 on tricky support messages. A "v means major release" rule got 1/8 and Jev got 8/8 on changelog titles (community, unverified: small hand-picked sets. The direction matches the docs).
• Combine: code settles deterministic states before the call ("Handle deterministic states without calling a model", docs' triage_ticket). A recall-tuned regex over-finds, Jev picks one, so the value is a verbatim match (Cookbook: Pre-parsed value extraction); Cookbook: Double-checking citations string-matches first. Code owns thresholds and weights (Composite scoring).

Embeddings, nearest neighbour, similarity search

• Beats Jev: the first pass over a whole corpus. The rerank cookbook calls BM25 or dense embeddings "fast search" that can "run on the whole pile"; judging every document means "millions of comparisons per query" (docs). No source gives embedding latency or price.
• Jev beats it: judging each shortlisted candidate. On 40 CLERC legal queries, BM25 shortlists were reranked with one Noul per pair (1,200 calls, $0.0645). Top-1 accuracy rose from 5% to 18% and top-10 from 38% to 62% (Cookbook: Re-ranking, docs, TypeSafe's run). Jev also labels relevance, contradiction and injected instructions per passage (Cookbook: Classifying RAG passages).
• Combine: retrieve with embeddings or BM25, then have Jev judge the top-k. The RAG cookbook keeps the top 12 by cosine similarity. Cookbook: Knowledge graph entity alignment judges 450 pre-generated candidate pairs; Jev does not generate them. The use-case map's "Replace or supplement embeddings" is TypeSafe's claim; its cookbooks only show "supplement".
• Silent: no source compares embeddings with Jev for dedup or clustering at scale. Using embeddings to block and Jev to confirm near-duplicates is (inferred).

Trained classifiers (classic ML, task-trained small models)

• Docs position: Jev "is not fine-tuned or LoRA-adapted with customer data". You adapt it through state, instructions, criteria (Models, aliases, pricing, rate limits, context). When you have labels, the documented route is Jev probabilities as features for a classical model: CatBoost in Cookbook: Autoresearch feature discovery, and "feed into a classical ML model" in How to build software with System One. Without labels, the docs suggest an ensemble of reasoning models to make them.
• Beats Jev (community): a task-trained specialist scored 99.7% decision accuracy against 83.6% for hosted Jev on one form task, at 7–9 ms local against 260–280 ms hosted (@trycua; unverified, narrow). The authors say the two measure "different boundaries". A 0.6B replica beat Jev on ViZDoom Basic (Field reports: independent evaluations, critiques, open replicas; unverified).
• Jev beats it: no training set or training loop, and new categories need only edited criteria (docs). @nedwize's tax-form classifier shipped without training: 0 wrong on 1,067 pages, compared with a Claude Sonnet classifier rather than a trained model (Measurements, access routes and open replicas; community, unverified).
• Combine: Jev answers become features and a small labelled set fits the weights. @razeden0 fitted weights over five Nouls on 1,000 labelled examples (Field reports: independent evaluations, critiques, open replicas; community, numbers unverified).
• Silent: no source measures a classic classifier (logistic regression, gradient boosting, fine-tuned BERT) against Jev on stable categories with many labels. Graduate-to-classifier is (inferred): launch on Jev, log answers and corrections, and once a labelled set proves a local model matches your decisions, move the stable high-volume categories to it; Jev keeps the long tail. Measurements, access routes and open replicas advises the same for self-hosted replicas (community).

Small fast chat LLMs with JSON mode

• Beats Jev: generation, including tool arguments, search queries and open extraction. In WindTunnel, Jev picked tools and Mercury 2.5 wrote the arguments (@0xidanlevin; verified direction, jaggedness §9). Cached input can be cheaper per token: @jpschroeder puts Jev at about 7x a DeepSeek V4.1 Flash cache read ($0.006) (Field reports: independent evaluations, critiques, open replicas; price verified). In TypeSafe's consistency run, Haiku 4.5 at temperature 0 was more repeatable than Jev (mean probability std dev 0.0012 against 0.0098) (Cookbook: Self-consistency — choices, docs).
• Jev beats it: in that same run, Jev took 114 ms and $0.000046 per call. The cheapest small-LLM setting (gpt-5.4-mini single-pick, t=0) took 826 ms and $0.000936. Jev also returns calibrated probabilities and cannot return a value outside your schema. The JSON-mode comparison is on Jev vs LLM JSON mode / structured outputs.
• Combine: the small LLM extracts or generates and Jev verifies. In the Cookbook: SDE cascade cascade, gpt-5.4-mini extracts, a battery of Jev Nouls scores P(wrong) for each field, and anything over 0.7 escalates. Or the LLM proposes candidates and Jev selects: "Select instead of generate" (The typesafe-ai agent skill and Claude Code plugin).

Frontier and reasoning LLMs

• Beats Jev: multi-hop reasoning, choosing the next action, writing prose or code, explanations for a reviewer, unfamiliar processes. TypeSafe itself lists chatbots, copilots and coding agents as LLM territory (System One Models). Jev is "not a drop-in replacement" for a coding agent's LLM (Jev with coding agents: not a drop-in for the LLM behind Claude Code, Cursor, Copilot).
• Jev beats it: narrow calls at volume. In TypeSafe's consistency run, gpt-5.5-reasoning took 12,978 ms and $0.041255 per call against Jev's 114 ms and $0.000046 (docs, TypeSafe's own run; TypeSafe says its published evals are "generally run from our laptops on the West Coast"). On judging, @LangChain reports quality-score variance 92–913x lower than GPT-5.6 Luna, Terra and Claude Sonnet 4.6 (community, verified in direction).
• Combine: use the frontier model as the escalation rung of a cascade. Alone, gpt-5.5-reasoning reached about 0.81 quality at about $0.10 per extraction. The cascade's frontier sits "up-and-left of every single model" (Cookbook: SDE cascade, internal TypeSafe results). They also generate labels.
• first-party caveat: in an agent loop, routing to a smaller model and back "can cost more because the context needs to be re-processed by the larger model" (Diogo Almeida, Field reports: independent evaluations, critiques, open replicas). Cascade independent items, not mid-session turns (inferred).

A human

• Beats Jev: high-stakes or irreversible actions, and cases where the signal isn't in the text. Diogo Almeida gives the example of sarcasm combined with a VIP customer: "this is when we escalate to a human" (first-party). Jev judges the text; it cannot forecast outcomes (Field reports: independent evaluations, critiques, open replicas, community).
• Jev beats it: volume and speed. TypeSafe's target is automation that is "99% machine-to-machine interactions and 1% human interaction" (AI primer: why calibrated decision models).
• Combine: use a confidence floor. In the docs' banking example, anything below 0.6 goes to a person, and a transfer needs more than 0.85 or else the user is asked to confirm (Confidence-gated routing). In Cookbook: Classification using confidence, a 0.9 cutoff split 60 filings in half: 90% correct above it, 40% below. Human labels are also how you tune those thresholds (Testing and evaluating a Jev workflow).

Decision table

The default composite: rules floor, Jev, escalate

1. Rules floor. Code settles everything exact and deterministic, filters the state, and builds the candidate list, using a regex, retrieval or an LLM proposer (How to build software with System One).
2. Jev. Many atomic questions in one request, each over a closed option set with an explicit other / none (Speculative fan-out, Choice questions).
3. Compose in code. Use weights or a classical model over the probabilities (Composite scoring).
4. Escalate on uncertainty. Send cases to a frontier model or a person: "if it's in the middle, then you do the next bigger model" (Diogo Almeida, first-party; Cookbook: SDE cascade is the documented form).
5. Graduate stable, high-volume slices to a trained model once your labels support it (inferred).

Related

• Jev vs LLM JSON mode / structured outputs: the Jev vs JSON-mode comparison this page does not repeat
• Jev 1.13 jaggedness: known failure modes: the official failure-mode list
• How to build software with System One: keep code in control
• Confidence-gated routing: floors and per-action thresholds
• Cookbook: SDE cascade: the documented cascade
• Field reports: independent evaluations, critiques, open replicas: community wins and failures
• Measurements, access routes and open replicas: community numbers
• Latent Space interview with Diogo Almeida (2026-09-21): first-party remarks on cascades, regex and fine-tuning

Sources

• raw/docs/model-jaggedness__jev-1.13.md (https://docs.typesafe.ai/model-jaggedness/jev-1.13): failure modes 1–9
• raw/docs/concepts__how-to-build-with-system-one.md (https://docs.typesafe.ai/concepts/how-to-build-with-system-one): "Use code when you can", classical-ML composition, routing on uncertainty, about 100 ms
• raw/docs/concepts__system-one.md, raw/site/blog-introducing-system-one.txt: scope and TypeSafe's LLM comparison
• raw/docs/introduction__machine-learning-primer.md: 99%/1% machine-to-human framing
• raw/docs/models.md: no fine-tuning, text only, aliases and pinning
• raw/docs/cookbooks__sde_cascade.md, cookbooks__pre_parsed_value_extraction_cookbook.md, cookbooks__rerank_typesafe.md, cookbooks__classifying_rag_passages.md, cookbooks__entity_alignment.md, cookbooks__autoresearch_feature_discovery.md, cookbooks__classification_using_confidence.md, cookbooks__consistency_choice_cookbook.md (https://docs.typesafe.ai/cookbooks)
• raw/docs/patterns__confidence-routing.md, raw/docs/concepts__use-case-map.md, raw/github/skills/skills/typesafe-ai/SKILL.md
• First-party: raw/community/latent-space-jev-diogo-almeida.md (Latent Space, 2026-09-21: regex, fine-tuning, cascade, human escalation); raw/community/diogo-almeida-coding-agent-notes.md (routing cost)
• Community: raw/nate/jev-shaped-problems.md (Nate B. Jones, https://unlock-ai.natebjones.com/guides/jev-shaped-problems); raw/x/trycua-2101437979180904640.md; raw/x/0xidanlevin-2100937437325205568.md; raw/x/jpschroeder-2100952337904767258.md; raw/x/LangChain-2101454284927959080.md; raw/x-repos/kyotofin__tax-doc-classifier.md; raw/community/archerhume-jevs-architecture-unmasked.md