---
title: "Head-to-head: Jev against other models and methods"
type: community
source_tier: community
tags: [head-to-head, benchmarks, cascades, cost, community]
created: 2026-09-23
updated: 2026-09-23
confidence: medium
sources:
  - raw/community/dev-to-bengreenberg-jev-vs-claude-who-wins-4mln.md
  - raw/community/primeline-cc-blog-typesafe-jev-pre-registered-test.md
  - raw/community/amankumar-ai-blogs-jev-measured.md
  - raw/community/blog-r6i-it-typesafe-jev-vs-agentic-loop-html.md
  - raw/community/langchain-com-blog-jev-agent-evals-langsmith.md
  - raw/community/dev-to-maximsaplin-typesafe-jev-played-chess-and-landed-next-to-reasoning-models-28ga.md
  - raw/community/dev-to-gabrielanhaia-jev-beat-gpt-luna-by-1-point-gpt-6-and-claude-wrote-the-answer-key-31.md
  - raw/community/medium-com-data-science-collective-i-tested-typesafes-jev-a-470-cheaper-decision-model-aga.md
  - raw/x/stas_sorokin_-2101994942818115738.md
  - raw/x-repos/stas4000__jev-papers.md
  - raw/x-repos/perixtar__jev-e2e.md
  - raw/x-repos/trungdq88__jev-tetris.md
  - raw/x-repos/4esv__jev-eval.md
  - raw/x-repos/QuicqDev__Jev-vs-ML.md
  - raw/x-repos/anessbelbati__jev-rerank-bench.md
  - raw/x-repos/zhuyansen__jev-search-rerank-eval.md
  - raw/x-repos/bartlomein__oko.md
  - raw/x-repos/fstandhartinger__jevbench.md
  - raw/x-repos/virajbhartiya__laya-vs-jev.md
  - raw/x/xunaoo-2101940075072459255.md
  - raw/x/OpenRouter-2101412965765529853.md
  - raw/x/vertr_ai-2102009557405089886.md
  - raw/x/vadimchoi-2101937206063780084.md
  - raw/x/Divine_machine-2102120832638050430.md
  - raw/x/isNickMa-2100566407524344225.md
  - raw/x/NicerInPerson-2102420798866694519.md
  - raw/x/iannuttall-2100884132272181594.md
  - raw/x/sl1ma4-2102008698767904981.md
  - raw/x/cruzex100-2102202098968666432.md
  - raw/x/KashyapVisharad-2102350346031149183.md
  - raw/x/b0dre-2102362736839512538.md
  - raw/x/techdrivenpetru-2101965334848663775.md
  - raw/x/svencreations-2102007851459748295.md
  - raw/x/voxmenthe-2102289186560589852.md
jev_version: "jev-1.13.0"
summary: "Builder-run comparisons of Jev against frontier, small, local and classic methods, including losses; cascade evidence and a per-decision cost ledger. One run each; re-measure."
---

# Head-to-head: Jev against other models and methods

> **TL;DR** Short input and crisp labels: Jev ties or beats small and frontier LLMs at a fraction of the cost and latency. Long or fuzzy input, a narrow benchmark some model was trained on, or a search that has to backtrack: it loses. Its biggest edge is **selective accuracy**: the confident answers are right, so the usual design is a cascade, not a swap. One builder's run per row; more in [[ideas/measurements]].

Numbers as each builder published them. **LOST** = Jev lost on accuracy or the stated goal. Patterns: [[ideas/patterns-data]] P15 judging, P17 rerank, P18 corpus labels, P24 games; [[ideas/patterns-interface]] P12 browser; [[ideas/patterns-agents]] P01 loops, P03 safety gates, P05 routing.

## vs frontier LLMs

| Task (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| 3-state policy gate, 102 hackathon entries × 3 runs ([Ben Greenberg](https://dev.to/bengreenberg/jev-vs-claude-who-wins-4mln)) | Jev Choice + 4 Nouls vs Claude Sonnet 5, high reasoning | 100.0% vs 99.0%; 378 vs 3,554 ms median; ECE 0.037 vs 0.058. Jev's 2 errors sat at 0.2-0.3 confidence, Sonnet's 3 at 0.9-1.0 | Jev on all three | His labels; skewed classes (77/7/18). His hand-coded rule over the same 4 Nouls fell to 94.1% with 6 false passes |
| Commit type (n=800) and note category (n=450), pre-registered ([Robin, primeline.cc](https://primeline.cc/blog/typesafe-jev-pre-registered-test)) | Jev vs Opus 5, GPT-5.6, Haiku 4.5 | Jev / Opus / GPT / Haiku: commits 65.8 / 63.5 / 59.5 / 54.6%; notes 90.7 / 86.9 / 92.7 / **97.8%** | Commits: Jev (vs Opus not significant). Notes: **LOST** (3rd) | One repo, labels partly Claude-written. Keeping each model's most-confident 30% of commits: Jev 89.6% vs Opus 80.8% |
| 1,000 arXiv papers, 24 topics ([@stas_sorokin_](https://x.com/stas_sorokin_/status/2101994942818115738), [repo](https://github.com/stas4000/jev-papers)) | Jev vs Opus 5 judge on 100 | Agreed 85/100 (95 in Jev's top 2); 98% agreement when Jev ≥ 0.90 (62 papers), 63% below; 57 ms vs 1,923 ms per paper | Latency, cost | Judge ≠ truth; Opus's first run returned nothing (reasoning used its budget) |
| Agent-trace pass/fail, 5 cases × 100 reps ([LangChain](https://www.langchain.com/blog/jev-agent-evals-langsmith)) | vs GPT-5.6 Luna, Terra, Claude Sonnet 4.6 | Agreement with human label: 100% vs Terra 99.8, Luna 96.4, Claude 80.0% | All | 5 cases; cost and variance rows in [[ideas/measurements]] |
| Product taxonomy descent, 50 items ([Sam Reghenzi](https://blog.r6i.it/typesafe-jev-vs-agentic-loop.html)) | Jev Choice per level vs gpt-5.2 agent + LLM judge | 1.38 vs 9.62 s mean; 3.18 vs 7.22 calls; 36/50 paths identical; of 14 others, 5 Jev, 4 agent, 4 undecidable, 1 both wrong | Latency, calls; accuracy tie | Old run 5 threads, new sequential; one annotator. Jev fails on level-1 branch picks it cannot undo |
| eBay E2E flow, 3 attempts each ([Jason Lu](https://github.com/perixtar/jev-e2e)) | vs GPT-5.6 Luna, Claude Sonnet 5 | 47.46 / 61.99 / 78.62 s; $0.006678 / $0.027704 / $0.406216 per case; UI choices 30/31, 32/32, 19/19 | Latency, cost; the only wrong choice was Jev's | Completed cases 1 / 2 / 1 |
| Chess, 80 games ([Maxim Saplin](https://dev.to/maximsaplin/typesafe-jev-played-chess-and-landed-next-to-reasoning-models-28ga)) | LLM Chess leaderboard | Elo 242.9 ± 117.5 (#59), beside o4-mini-medium 240.3 ($1.11 and 41.5 min per game); Jev $0.0015 and 35.6 s per game; every game completed | Cost, latency; accuracy tie | Wide Elo interval |
| Browser-agent step ([@xunaoo](https://x.com/xunaoo/status/2101940075072459255)) | vs his earlier LLM step | 2 s → 0.2 s; 10 model calls → 2; accuracy **6 points lower** | Latency; **LOST** accuracy | No sample size |
| Short claims, no data | | Judging: ">5x faster than the next fastest" ([OpenRouter Ori Eval](https://x.com/OpenRouter/status/2101412965765529853)); 154 vs 860 ms on 200 synthetic cases ([@vertr_ai](https://x.com/vertr_ai/status/2102009557405089886)); desktop agent ~10x faster than GPT-6-Astra computer use ([@Divine_machine](https://x.com/Divine_machine/status/2102120832638050430)); wiki recall 10x cheaper, 40x faster than Sonnet ([@NicerInPerson](https://x.com/NicerInPerson/status/2102420798866694519)); safety monitor faster than Gemini ([@isNickMa](https://x.com/isNickMa/status/2100566407524344225)) | Speed | `unverified`; speed only |
| 200 decisions ([Manjunath Janardhan](https://medium.com/data-science-collective/i-tested-typesafes-jev-a-470-cheaper-decision-model-against-claude-gpt-6-kimi-minimax-and-d36ed152e861)) | vs Claude, GPT-6, Kimi, MiniMax, DeepSeek | Headline: "Only Two Beat It" | **LOST** to two | Paywalled; the two not named in capture |

## vs small and fast LLMs

| Task (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| 4 public sets × 300 ([Aman Kumar](https://amankumar.ai/blogs/jev-measured)) | Jev / gpt-5.4-mini / gpt-5.6-luna | Enron 98.7 / 97.7 / 98.0; SST-2 95.7 / 92.7 / 93.0; AG News 91.3 / 88.3 / 89.7; Banking77 **76.0** / 78.7 / 81.7. Median 0.8-0.9 s vs 1.4-5.0 s | 3 of 4 sets; **LOST** Banking77 | At 100 in flight a few calls per thousand took 10-35 s |
| Same author, production | vs his current small LLM | Whole-document "carries a value for one of 135 rows": 68.6%, worse; a general LLM failed the same way | **LOST** | Long input |
| Phishing, 2,000 emails (relayed by Kumar) | vs Claude Haiku 4.5 | 62.6% vs 81.3%; 5 yes/no signals + logistic regression 95%, Haiku on those signals 93% | **LOST**, then won once split | Original not captured |
| Real-time Tetris, seed 42 ([Tony Dinh](https://github.com/trungdq88/jev-tetris)) | vs Claude Haiku 4.5 | Won all 3 modes; ~220 vs ~750 ms per move; ~20x cheaper per move | All | One run per mode |
| Same, Gemini 3.8 Flash | | Real time: Jev won (Gemini missed 8 deadlines). No clock: Gemini 0.32 lines per piece vs 0.25 at ~17x the cost per move | **LOST** with no clock | One seed |
| Content tagging ([@iannuttall](https://x.com/iannuttall/status/2100884132272181594)) | vs GLM 4.7 Flash | 50x faster, "no failures" | Latency | No accuracy published |
| Who answers in a group chat ([@vadimchoi](https://x.com/vadimchoi/status/2101937206063780084)) | vs LLM orchestrator | 0.29 s median, $0.00002 vs 4-7 s, $0.00046 per message | Latency, cost | No accuracy published |
| fx command safety reviewer (Vercel, relayed by [Gabriel Anhaia](https://dev.to/gabrielanhaia/jev-beat-gpt-luna-by-1-point-gpt-6-and-claude-wrote-the-answer-key-314k)) | vs GPT-5.6 Luna | "~5-18x faster and more accurate" | Claimed both | No dataset or case count published |

## vs local or open models

| Task (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| 5 labelled tasks × 300 ([4esv](https://github.com/4esv/jev-eval)) | Jev / GPT-5.6 Terra / open-jev 435M / Kev-0.8B / Laya 421M | CLINC-150 0.897 / — / 0.610 / 0.643 / 0.497. Banking77 **0.780** / 0.847 / **0.873** / 0.770 / 0.370. SST-5 0.570 / 0.593 / 0.560 | CLINC: Jev. **LOST** Banking77 and SST-5 | open-jev and Kev trained on Banking77 and SST-5. Only Jev's latency stayed flat from 2 to 151 options (0.17 → 0.20 s). open-jev was best calibrated on 4 of 5 |
| Snake, 30 s ([@sl1ma4](https://x.com/sl1ma4/status/2102008698767904981)) | vs Laya, local | 86.5 vs 3.2 decisions/s; score 46 vs 1 | **LOST** | Speed-bound game; Laya's window 512-1024 tokens |
| Clash Royale, 1 match ([@KashyapVisharad](https://x.com/KashyapVisharad/status/2102350346031149183)) | vs Laya on a Mac | Laya won | **LOST** | n = 1 |
| Tetris ([Tony Dinh](https://github.com/trungdq88/jev-tetris)) | vs Laya, 4-core CPU | Jev won at 0:14: 7 lines vs 0; 232 vs 614 ms per move | All | Laya near-flat; prefers first-listed options |
| Smoke test ([@cruzex100](https://x.com/cruzex100/status/2102202098968666432)) | vs Laya | Soft accuracy 0.580 vs 0.471; ECE 0.144 vs 0.213; ~710 vs ~30-40 ms; ~$0.0004 vs ~$0 per decision | Accuracy, calibration; **LOST** latency, cost | Task not stated |
| Movie search ([@b0dre](https://x.com/b0dre/status/2102362736839512538)) | vs Laya on a Radeon Pro 5500 XT | 392 ms local vs 676 ms API, "same answer" | **LOST** latency | One query |
| Reranking, NevIR negation ([anessbelbati](https://github.com/anessbelbati/jev-rerank-bench)) | vs Open-Jev 9B, Laya | nDCG: Jev yes/no per pair beats Open-Jev 9B by 7.0 points; Laya ≈ BM25. Negation pairs: Open-Jev 9B **77%** vs Jev 71%, Laya 36% | nDCG: Jev. **LOST** negation to 9B | Open-Jev 9B: 19.1 s per query |
| JevBench, 534 frozen decisions ([fstandhartinger](https://github.com/fstandhartinger/jevbench)) | 48 ranked systems | Jev 1.13.0 #1 at 74.4; SemIf 73.1; djev 73.0 | Composite | Mixes accuracy, calibration, speed, cost; hard items written by Opus 5 and GPT-5.6 Sol; self-hosted latency ×2 + 0.15 s by assumption |
| Pre-tool-use gate ([@techdrivenpetru](https://x.com/techdrivenpetru/status/2101965334848663775)) | Jev vs local DeBERTa-large | Local "just as fast, just as good"; admits misfires | Claimed tie | `unverified`; no numbers |

Trust notes: planner-assisted game scores measure the planner plus the model ([laya-vs-jev](https://github.com/virajbhartiya/laya-vs-jev)). [@svencreations](https://x.com/svencreations/status/2102007851459748295) tried several "Jev killers": "Most weren't even close." Replica notes: [[ideas/measurements]].

## vs classic methods

| Task (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| 8 datasets, 3 seeds ([QuicqDev](https://github.com/QuicqDev/Jev-vs-ML)) | vs 11 trained classical pipelines | IMDb 96.3% vs 88.4% balanced accuracy; classical models led on all 4 tabular sets | Text: Jev. **LOST** tabular | Threshold-adjusted rows use labelled data |
| Rerank BM25 top 30, 8 datasets, 1,617 queries ([anessbelbati](https://github.com/anessbelbati/jev-rerank-bench)) | vs Cohere Rerank 4 Pro, zerank-2, DeepSeek | nDCG@10 0.692 / 0.691 / 0.682 / 0.682; BM25 0.486. 422 vs 844 ms; $0.45 vs $2.51 per 1,000 queries | Tie; latency. Per-query weighting: **LOST** (0.738 vs 0.756) | Reversing passage order changed Jev's top pick on 24.7% of queries |
| Skill search, 164 zh/en queries ([zhuyansen](https://github.com/zhuyansen/jev-search-rerank-eval)) | Jev rerank vs bge-m3 embeddings | Rerank alone +0.012 nDCG (CI crosses 0), **−0.028** on labels Jev had no part in; RRF fusion +0.090 (+0.064) | **LOST** alone; fusion best | Two LLM judges; 30 pairs hand-checked |
| Code retrieval, 345 tasks ([Oko](https://github.com/bartlomein/oko)) | vs BM25, embeddings, RepoMap | MRR 0.39 vs 0.24 best; needed code within 8k tokens 0.48 vs BM25 0.21; top-20 recall 0.64 vs **0.70** for Qwen3-Embedding-8B | Ranking; **LOST** recall | Builder's own run |
| Code search ([@voxmenthe](https://x.com/voxmenthe/status/2102289186560589852)) | BM25 + Jev | "Didn't work well enough"; rebuilt with 4 lexical channels before Jev | **LOST** in the naive form | His claim of ~$0.0015 per query for ~18k tokens does not reconcile: 18k × $0.042/Mtok ≈ $0.00076 |
| Chess vs Komodo Dragon, levels 1-3 ([Maxim Saplin](https://dev.to/maximsaplin/typesafe-jev-played-chess-and-landed-next-to-reasoning-models-28ga)) | vs chess engine | 50% draws at every level, 0 wins | **LOST** | 10 games per level |
| Spam, 19,500 emails (relayed by Kumar) | vs logistic regression trained on 15,000 | 98.3% zero-shot, level | Tie with no training | Original not captured |

## Cascades (Jev plus a bigger model)

- **Trust the ends, not the middle** (Kumar): confident answers were right 90-100% of the time, the middle was a coin flip. Drop line = half the lowest P(yes) any known positive got: page gates skipped 25-60% of pages, losing nothing. Email had no lossless line: under 0.03 dropped 78% of non-requests and 5 of 231 real ones. Pattern: [[ideas/patterns-data]] P16, [[ideas/patterns-agents]] P06.
- **Automate above a threshold** (Greenberg): at 0.5, 98% of decisions ran automatically at 100% accuracy; the other 2% went to a human.
- **Cheap model sorts, expensive model audits what it flags** (Sorokin): 14 of 15 disagreements had Jev confidence under 0.90.
- **Triage first, agent on the leftovers** (Reghenzi). Neither his geometric-mean path score nor Jev's confidence caught the wrong branch picks. His fix is one extra verification question, at +31% latency.
- **Fuse, don't replace** (zhuyansen): Jev's rank added to embedding ranks gave the best system.
- **What did not help**: in JevBench, confidence cascades, committees and best-of-n: "None changed the ranked board.". Cascading within Jev (batch prune, then 8 pairs) scored 0.674, below one 30-question call at 0.692 (anessbelbati). Two-stage narrowing was 12.9 points worse on overlapping categories (Robin). One narrow Noul added before the main question fixed 23 of 53 errors (Robin).

## Published cost per decision

Per-decision figures are the source's, or ÷ where it publishes both numbers. Price is `verified`: $0.042 per million input tokens, output free ([[reference/models-and-pricing]]).

| Workload | Volume | Total | Per decision | Source |
|---|---|---|---|---|
| Paper topics | 1,000 papers, 1,392,141 tokens | $0.0585 (reconciles at list price) | $0.000058; Opus judge $0.00894 | Sorokin |
| Policy gate, projected | 10,000 | $2.27 vs Sonnet $129.74 | ÷ $0.000227 vs $0.0130 | Greenberg |
| Pre-registered tests | ~9,750 calls | ~$0.38 | ÷ ~$0.000039 | Robin |
| JevBench | 534 decisions, 950 tokens average | $0.0399 per 1,000 | $0.0000399 | fstandhartinger |
| Chess | 80 games | ≈ $0.12 | $0.0015 per game, ~$0.00002 per move | Saplin |
| Relevance labels | 9,831 pairs | $0.069 vs Haiku $2.37 | ÷ $0.0000070 vs $0.00024 | zhuyansen |
| Rerank | 328 queries | $0.066 | ~$0.0002 | zhuyansen |
| Classical-ML benchmark | 38,922 attempts | ~$4.19, an estimate, not an invoice | ÷ ~$0.00011 | QuicqDev |
| Tetris game | 150 pieces, ~530k tokens | "about two cents" | ÷ ~$0.00013 per move | Dinh |
| Email triage / page gate / admin gate | per 1,000 | 15¢ vs $120 agent; 11¢ vs $1.50; 6¢ vs $3.50 | as stated | Kumar |

## How to read these

- **All builder-run**, no shared benchmark; tasks, labels, reasoning settings and concurrency differ. TypeSafe publishes no benchmark table ([[entities/blog-antibenchmaxxing]]). Its own board is [[concepts/workflow-evals]], graded against labels from GPT-6 Astra and Claude Fable 5.1. On that board Jev scores 67.8% against 74.1% for GPT-5.6 Sol and is 17.3 points behind on invoice processing (Anhaia, Kumar).
- **Labels decide the result.** A model trained on the dataset won it (4esv); on labels Jev had no part in, its lead vanished (zhuyansen); an LLM judge measures agreement, not truth (Sorokin).
- **Losses cluster.** Jev lost on long input, whole-document reads, benchmarks another model trained on, tabular data, search that must backtrack, negation against a 9B, and games where raw speed wins. See [[concepts/jaggedness-jev-1-13]] and [[ideas/failure-reports]].
- **Compare selective accuracy, not only raw accuracy.** A Choice `confidence` is the top probability rescaled by option count (Robin, citing Stanislav Yurin; consistent with the documented approximation in [[concepts/confidence]]). Don't reuse a threshold across questions with different option counts.

## Related

- [[ideas/measurements]]: numbers not repeated here; [[ideas/failure-reports]]: what broke
- [[ideas/builds-agents]], [[ideas/builds-apps]], [[ideas/tools-and-integrations]], [[ideas/patterns-emerging]]
- [[guides/testing-and-evaluation]]: run your own comparison. [[patterns/confidence-routing]]: the official cascade.

## Sources

Links inline; raw captures listed in the frontmatter (private repo).