~/wiki/ideas
Head-to-head: Jev against other models and methods
TL;DR Short input and crisp labels: Jev ties or beats small and frontier LLMs at a fraction of the cost and latency. Long or fuzzy input, a narrow benchmark some model was trained on, or a search that has to backtrack: it loses. Its biggest edge is selective accuracy: the confident answers are right, so the usual design is a cascade, not a swap. One builder's run per row; more in Measurements, access routes and open replicas.
Numbers as each builder published them. LOST = Jev lost on accuracy or the stated goal. Patterns: Patterns: judging, search, documents, real-time and markets P15 judging, P17 rerank, P18 corpus labels, P24 games; Patterns: browser, computer use, voice and product UI P12 browser; Patterns: agent internals, context and coding agents P01 loops, P03 safety gates, P05 routing.
vs frontier LLMs
| Task (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| 3-state policy gate, 102 hackathon entries × 3 runs (Ben Greenberg) | Jev Choice + 4 Nouls vs Claude Sonnet 5, high reasoning | 100.0% vs 99.0%; 378 vs 3,554 ms median; ECE 0.037 vs 0.058. Jev's 2 errors sat at 0.2-0.3 confidence, Sonnet's 3 at 0.9-1.0 | Jev on all three | His labels; skewed classes (77/7/18). His hand-coded rule over the same 4 Nouls fell to 94.1% with 6 false passes |
| Commit type (n=800) and note category (n=450), pre-registered (Robin, primeline.cc) | Jev vs Opus 5, GPT-5.6, Haiku 4.5 | Jev / Opus / GPT / Haiku: commits 65.8 / 63.5 / 59.5 / 54.6%; notes 90.7 / 86.9 / 92.7 / 97.8% | Commits: Jev (vs Opus not significant). Notes: LOST (3rd) | One repo, labels partly Claude-written. Keeping each model's most-confident 30% of commits: Jev 89.6% vs Opus 80.8% |
| 1,000 arXiv papers, 24 topics (@stas_sorokin_, repo) | Jev vs Opus 5 judge on 100 | Agreed 85/100 (95 in Jev's top 2); 98% agreement when Jev ≥ 0.90 (62 papers), 63% below; 57 ms vs 1,923 ms per paper | Latency, cost | Judge ≠ truth; Opus's first run returned nothing (reasoning used its budget) |
| Agent-trace pass/fail, 5 cases × 100 reps (LangChain) | vs GPT-5.6 Luna, Terra, Claude Sonnet 4.6 | Agreement with human label: 100% vs Terra 99.8, Luna 96.4, Claude 80.0% | All | 5 cases; cost and variance rows in Measurements, access routes and open replicas |
| Product taxonomy descent, 50 items (Sam Reghenzi) | Jev Choice per level vs gpt-5.2 agent + LLM judge | 1.38 vs 9.62 s mean; 3.18 vs 7.22 calls; 36/50 paths identical; of 14 others, 5 Jev, 4 agent, 4 undecidable, 1 both wrong | Latency, calls; accuracy tie | Old run 5 threads, new sequential; one annotator. Jev fails on level-1 branch picks it cannot undo |
| eBay E2E flow, 3 attempts each (Jason Lu) | vs GPT-5.6 Luna, Claude Sonnet 5 | 47.46 / 61.99 / 78.62 s; $0.006678 / $0.027704 / $0.406216 per case; UI choices 30/31, 32/32, 19/19 | Latency, cost; the only wrong choice was Jev's | Completed cases 1 / 2 / 1 |
| Chess, 80 games (Maxim Saplin) | LLM Chess leaderboard | Elo 242.9 ± 117.5 (#59), beside o4-mini-medium 240.3 ($1.11 and 41.5 min per game); Jev $0.0015 and 35.6 s per game; every game completed | Cost, latency; accuracy tie | Wide Elo interval |
| Browser-agent step (@xunaoo) | vs his earlier LLM step | 2 s → 0.2 s; 10 model calls → 2; accuracy 6 points lower | Latency; LOST accuracy | No sample size |
| Short claims, no data | Judging: ">5x faster than the next fastest" (OpenRouter Ori Eval); 154 vs 860 ms on 200 synthetic cases (@vertr_ai); desktop agent ~10x faster than GPT-6-Astra computer use (@Divine_machine); wiki recall 10x cheaper, 40x faster than Sonnet (@NicerInPerson); safety monitor faster than Gemini (@isNickMa) | Speed | unverified; speed only |
|
| 200 decisions (Manjunath Janardhan) | vs Claude, GPT-6, Kimi, MiniMax, DeepSeek | Headline: "Only Two Beat It" | LOST to two | Paywalled; the two not named in capture |
vs small and fast LLMs
| Task (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| 4 public sets × 300 (Aman Kumar) | Jev / gpt-5.4-mini / gpt-5.6-luna | Enron 98.7 / 97.7 / 98.0; SST-2 95.7 / 92.7 / 93.0; AG News 91.3 / 88.3 / 89.7; Banking77 76.0 / 78.7 / 81.7. Median 0.8-0.9 s vs 1.4-5.0 s | 3 of 4 sets; LOST Banking77 | At 100 in flight a few calls per thousand took 10-35 s |
| Same author, production | vs his current small LLM | Whole-document "carries a value for one of 135 rows": 68.6%, worse; a general LLM failed the same way | LOST | Long input |
| Phishing, 2,000 emails (relayed by Kumar) | vs Claude Haiku 4.5 | 62.6% vs 81.3%; 5 yes/no signals + logistic regression 95%, Haiku on those signals 93% | LOST, then won once split | Original not captured |
| Real-time Tetris, seed 42 (Tony Dinh) | vs Claude Haiku 4.5 | Won all 3 modes; ~220 vs ~750 ms per move; ~20x cheaper per move | All | One run per mode |
| Same, Gemini 3.8 Flash | Real time: Jev won (Gemini missed 8 deadlines). No clock: Gemini 0.32 lines per piece vs 0.25 at ~17x the cost per move | LOST with no clock | One seed | |
| Content tagging (@iannuttall) | vs GLM 4.7 Flash | 50x faster, "no failures" | Latency | No accuracy published |
| Who answers in a group chat (@vadimchoi) | vs LLM orchestrator | 0.29 s median, $0.00002 vs 4-7 s, $0.00046 per message | Latency, cost | No accuracy published |
| fx command safety reviewer (Vercel, relayed by Gabriel Anhaia) | vs GPT-5.6 Luna | "~5-18x faster and more accurate" | Claimed both | No dataset or case count published |
vs local or open models
| Task (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| 5 labelled tasks × 300 (4esv) | Jev / GPT-5.6 Terra / open-jev 435M / Kev-0.8B / Laya 421M | CLINC-150 0.897 / — / 0.610 / 0.643 / 0.497. Banking77 0.780 / 0.847 / 0.873 / 0.770 / 0.370. SST-5 0.570 / 0.593 / 0.560 | CLINC: Jev. LOST Banking77 and SST-5 | open-jev and Kev trained on Banking77 and SST-5. Only Jev's latency stayed flat from 2 to 151 options (0.17 → 0.20 s). open-jev was best calibrated on 4 of 5 |
| Snake, 30 s (@sl1ma4) | vs Laya, local | 86.5 vs 3.2 decisions/s; score 46 vs 1 | LOST | Speed-bound game; Laya's window 512-1024 tokens |
| Clash Royale, 1 match (@KashyapVisharad) | vs Laya on a Mac | Laya won | LOST | n = 1 |
| Tetris (Tony Dinh) | vs Laya, 4-core CPU | Jev won at 0:14: 7 lines vs 0; 232 vs 614 ms per move | All | Laya near-flat; prefers first-listed options |
| Smoke test (@cruzex100) | vs Laya | Soft accuracy 0.580 vs 0.471; ECE 0.144 vs 0.213; ~710 vs ~30-40 ms; ~$0.0004 vs ~$0 per decision | Accuracy, calibration; LOST latency, cost | Task not stated |
| Movie search (@b0dre) | vs Laya on a Radeon Pro 5500 XT | 392 ms local vs 676 ms API, "same answer" | LOST latency | One query |
| Reranking, NevIR negation (anessbelbati) | vs Open-Jev 9B, Laya | nDCG: Jev yes/no per pair beats Open-Jev 9B by 7.0 points; Laya ≈ BM25. Negation pairs: Open-Jev 9B 77% vs Jev 71%, Laya 36% | nDCG: Jev. LOST negation to 9B | Open-Jev 9B: 19.1 s per query |
| JevBench, 534 frozen decisions (fstandhartinger) | 48 ranked systems | Jev 1.13.0 #1 at 74.4; SemIf 73.1; djev 73.0 | Composite | Mixes accuracy, calibration, speed, cost; hard items written by Opus 5 and GPT-5.6 Sol; self-hosted latency ×2 + 0.15 s by assumption |
| Pre-tool-use gate (@techdrivenpetru) | Jev vs local DeBERTa-large | Local "just as fast, just as good"; admits misfires | Claimed tie | unverified; no numbers |
Trust notes: planner-assisted game scores measure the planner plus the model (laya-vs-jev). @svencreations tried several "Jev killers": "Most weren't even close." Replica notes: Measurements, access routes and open replicas.
vs classic methods
| Task (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| 8 datasets, 3 seeds (QuicqDev) | vs 11 trained classical pipelines | IMDb 96.3% vs 88.4% balanced accuracy; classical models led on all 4 tabular sets | Text: Jev. LOST tabular | Threshold-adjusted rows use labelled data |
| Rerank BM25 top 30, 8 datasets, 1,617 queries (anessbelbati) | vs Cohere Rerank 4 Pro, zerank-2, DeepSeek | nDCG@10 0.692 / 0.691 / 0.682 / 0.682; BM25 0.486. 422 vs 844 ms; $0.45 vs $2.51 per 1,000 queries | Tie; latency. Per-query weighting: LOST (0.738 vs 0.756) | Reversing passage order changed Jev's top pick on 24.7% of queries |
| Skill search, 164 zh/en queries (zhuyansen) | Jev rerank vs bge-m3 embeddings | Rerank alone +0.012 nDCG (CI crosses 0), −0.028 on labels Jev had no part in; RRF fusion +0.090 (+0.064) | LOST alone; fusion best | Two LLM judges; 30 pairs hand-checked |
| Code retrieval, 345 tasks (Oko) | vs BM25, embeddings, RepoMap | MRR 0.39 vs 0.24 best; needed code within 8k tokens 0.48 vs BM25 0.21; top-20 recall 0.64 vs 0.70 for Qwen3-Embedding-8B | Ranking; LOST recall | Builder's own run |
| Code search (@voxmenthe) | BM25 + Jev | "Didn't work well enough"; rebuilt with 4 lexical channels before Jev | LOST in the naive form | His claim of ~$0.0015 per query for ~18k tokens does not reconcile: 18k × $0.042/Mtok ≈ $0.00076 |
| Chess vs Komodo Dragon, levels 1-3 (Maxim Saplin) | vs chess engine | 50% draws at every level, 0 wins | LOST | 10 games per level |
| Spam, 19,500 emails (relayed by Kumar) | vs logistic regression trained on 15,000 | 98.3% zero-shot, level | Tie with no training | Original not captured |
Cascades (Jev plus a bigger model)
- Trust the ends, not the middle (Kumar): confident answers were right 90-100% of the time, the middle was a coin flip. Drop line = half the lowest P(yes) any known positive got: page gates skipped 25-60% of pages, losing nothing. Email had no lossless line: under 0.03 dropped 78% of non-requests and 5 of 231 real ones. Pattern: Patterns: judging, search, documents, real-time and markets P16, Patterns: agent internals, context and coding agents P06.
- Automate above a threshold (Greenberg): at 0.5, 98% of decisions ran automatically at 100% accuracy; the other 2% went to a human.
- Cheap model sorts, expensive model audits what it flags (Sorokin): 14 of 15 disagreements had Jev confidence under 0.90.
- Triage first, agent on the leftovers (Reghenzi). Neither his geometric-mean path score nor Jev's confidence caught the wrong branch picks. His fix is one extra verification question, at +31% latency.
- Fuse, don't replace (zhuyansen): Jev's rank added to embedding ranks gave the best system.
- What did not help: in JevBench, confidence cascades, committees and best-of-n: "None changed the ranked board.". Cascading within Jev (batch prune, then 8 pairs) scored 0.674, below one 30-question call at 0.692 (anessbelbati). Two-stage narrowing was 12.9 points worse on overlapping categories (Robin). One narrow Noul added before the main question fixed 23 of 53 errors (Robin).
Published cost per decision
Per-decision figures are the source's, or ÷ where it publishes both numbers. Price is verified: $0.042 per million input tokens, output free (Models, aliases, pricing, rate limits, context).
| Workload | Volume | Total | Per decision | Source |
|---|---|---|---|---|
| Paper topics | 1,000 papers, 1,392,141 tokens | $0.0585 (reconciles at list price) | $0.000058; Opus judge $0.00894 | Sorokin |
| Policy gate, projected | 10,000 | $2.27 vs Sonnet $129.74 | ÷ $0.000227 vs $0.0130 | Greenberg |
| Pre-registered tests | ~9,750 calls | ~$0.38 | ÷ ~$0.000039 | Robin |
| JevBench | 534 decisions, 950 tokens average | $0.0399 per 1,000 | $0.0000399 | fstandhartinger |
| Chess | 80 games | ≈ $0.12 | $0.0015 per game, ~$0.00002 per move | Saplin |
| Relevance labels | 9,831 pairs | $0.069 vs Haiku $2.37 | ÷ $0.0000070 vs $0.00024 | zhuyansen |
| Rerank | 328 queries | $0.066 | ~$0.0002 | zhuyansen |
| Classical-ML benchmark | 38,922 attempts | ~$4.19, an estimate, not an invoice | ÷ ~$0.00011 | QuicqDev |
| Tetris game | 150 pieces, ~530k tokens | "about two cents" | ÷ ~$0.00013 per move | Dinh |
| Email triage / page gate / admin gate | per 1,000 | 15¢ vs $120 agent; 11¢ vs $1.50; 6¢ vs $3.50 | as stated | Kumar |
How to read these
- All builder-run, no shared benchmark; tasks, labels, reasoning settings and concurrency differ. TypeSafe publishes no benchmark table (Blog: Lies, Damned Lies, and Benchmarks). Its own board is Workflow evals: how TypeSafe measures Jev, graded against labels from GPT-6 Astra and Claude Fable 5.1. On that board Jev scores 67.8% against 74.1% for GPT-5.6 Sol and is 17.3 points behind on invoice processing (Anhaia, Kumar).
- Labels decide the result. A model trained on the dataset won it (4esv); on labels Jev had no part in, its lead vanished (zhuyansen); an LLM judge measures agreement, not truth (Sorokin).
- Losses cluster. Jev lost on long input, whole-document reads, benchmarks another model trained on, tabular data, search that must backtrack, negation against a 9B, and games where raw speed wins. See Jev 1.13 jaggedness: known failure modes and Failure reports: where Jev broke, lost, or was the wrong tool.
- Compare selective accuracy, not only raw accuracy. A Choice
confidenceis the top probability rescaled by option count (Robin, citing Stanislav Yurin; consistent with the documented approximation in Confidence vs probability). Don't reuse a threshold across questions with different option counts.
Related
- Measurements, access routes and open replicas: numbers not repeated here; Failure reports: where Jev broke, lost, or was the wrong tool: what broke
- Builds: agents, coding tools, browser and interface, Builds: data, content, business, games and devices, Tools and integrations: MCP servers, skills, clients, gateways and libraries, Patterns: gates, simulation, personas and other shapes (P38+)
- Testing and evaluating a Jev workflow: run your own comparison. Confidence-gated routing: the official cascade.
Sources
Links inline; raw captures listed in the frontmatter (private repo).