---
title: "Head-to-head: Jev vs rival decision models"
type: community
source_tier: community
tags: [head-to-head, benchmarks, rival-models, advising]
created: 2026-10-24
updated: 2026-10-24
confidence: medium
sources: "67 files, listed under this page in https://jevwiki.ai/index.json (raw/ paths map to their URLs in https://jevwiki.ai/raw/MANIFEST.json); links are inline in the body"
jev_version: "jev-1.14.0"
summary: "Builder-run comparisons of Jev with other vendors' decision models (Clef, Perplexity, GLiDE, Mercury Decide, StartLux and others): accuracy, calibration, cost, latency."
---

# Head-to-head: Jev vs rival decision models

> **TL;DR** Builder-run comparisons of Jev with other vendors' decision models; numbers as each builder published them, one run each; **LOST** = Jev lost on accuracy or the stated goal. Split from [[ideas/head-to-head-small-models]] on 2026-10-24 (rows unchanged); the vendors' own numbers live on [[ideas/rival-decision-models]], OpenAI's Decisions API on [[ideas/head-to-head-decisions-api]]. Rival decision APIs (Clef, Perplexity, GLiDE; added 2026-10-04, more Clef runs 2026-10-05): against Clef, Jev was usually faster, cheaper and better calibrated; accuracy split (Clef ahead on a rule-fixed Japanese set and a 1,565-email set, Jev ahead or tied on agent tasks and a 260-item Japanese set). Clef's `confidence` runs low, so a Jev threshold does not carry over. On a 1,999-item public board (s1bench) Jev led Clef-flash by under two points; Clef-flash was faster and better calibrated (added 2026-10-06). Added 2026-10-09: Vals AI found Inception's Mercury Decide level with Jev on claim checks, ahead on legal entailment and cheaper per single question, but slower and dearer when many questions share one document; on a slide-visual task every model, Jev included, scored far lower on real user prompts than on synthetic ones.

## Comparisons

Decision APIs from other vendors launched 2026-10-01 (Cloudflare's Clef and Clef-flash; Perplexity's model, name not given in these posts), a self-reported rival (GLiDE) and, from 2026-10-08, Inception's Mercury Decide, and from 2026-10-12 Subquadratic's SubQ Snap (preview, long context, reads images). OpenAI's Decisions API appears here only inside multi-model runs; Jev-vs-Decisions runs live on [[ideas/head-to-head-decisions-api]]. None is Jev; vendor facts and prices are on [[ideas/rival-decision-models]]; replicas on [[ideas/open-replicas]]. Injection-set runs: [[ideas/head-to-head-security]].

| Task (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| Business email category (10), 1,565 emails, 6,260 calls from one US cloud server ([@nikhilmudholkar](https://x.com/nikhilmudholkar/status/2106000029320400984) thread, [latency and cost post](https://x.com/nikhilmudholkar/status/2106001371539595424) and [rerun](https://x.com/nikhilmudholkar/status/2106470593310990699), 2026-10-02 and 03) | Jev vs Perplexity's decision model, Clef, Clef-flash | Accuracy Clef 97.7% vs Jev 96.4%; the thread says the cheapest model (Perplexity) was also the most accurate (its figure is not in our capture). Typical latency 0.26 / 0.29 / 0.34 / 0.64 s (Perplexity, Jev, Clef-flash, Clef); $0.06 / $0.08 / $0.15 / $0.41 per 1,000 emails. Clef at ~75% stated confidence was right 98% (underconfident) | **LOST** accuracy to Clef and Perplexity; cost and latency to Perplexity | `unverified`: thread posts only, most of the 8-post thread not captured |
| Next page to recommend (one of 7), the same 113 synthetic Japanese site visitors and rule-fixed answers as the GPT-6 Luna run ([[ideas/head-to-head]]), `state` and questions sent unchanged; predictions pre-registered with hashes ([@RenDaichi147380](https://x.com/RenDaichi147380/status/2106683439709589560) and its X article, run 2026-10-04; 1,064 calls) | Jev vs Clef, Clef-flash (Workers AI). OpenAI's Decisions API still returned 403 | Accuracy 67.3 / **83.2** / 45.1% (English: Clef 72.6, Jev 67.3); easy cases 76.8 vs 98.2%, hard 45.5 vs 59.1% (Jev vs Clef); Clef-flash picked `works` on 67 of 113 (right on 32). Clef's `confidence` never reached 0.9 (max 0.893, median 0.34), yet its 0.4-0.6 band was 95.1% right; Jev's 0.9-1.0 band averaged 95.3% and was 95.6% right. Top 40% by confidence: Jev 95.6 vs Clef 97.8%; top 60%: 80.9 vs 97.1%. Re-cut: Clef ≥ 0.30 kept 61.9% at 97.1% vs Jev ≥ 0.9 37.2% at 97.6%. 15 repeats of the 8 items nearest 0.9: Jev's answers never changed but `confidence` moved (mean range 0.07) and crossed 0.9 on 6 of 8; Clef and Clef-flash identical to 4 decimals. Jev on 2026-10-01 vs 10-04: 9 of 113 answers differed, same accuracy. p50 from a Japanese home line 0.12 / 0.51 / 0.27 s (p95 0.22 / 0.87 / 0.65). Junk `state`: Clef 0.09, Clef-flash 0.19 (Jev 0.68-0.71 in the earlier run), but 38 of Clef's 113 real answers sat below junk's 0.27. Jobseeker Noul at 0.5 (226 items, both languages): 93.8 / 98.7 / 89.4%. Clef ¥27.9 per 1,000 decisions at ¥150/$ | **LOST** accuracy and coverage at equal precision to Clef; Jev fastest | Answers fixed by 7 written rules, which suits an LLM-based model; rules this explicit are an if-statement's job (author). One 7-option question. Clef's `confidence` formula is unpublished and fits neither Strands Decider's published one nor margin or entropy (author). The re-cut moved with language (English Clef: 0.30 gave 94.2%, 0.50 98.0%). Cloudflare's own latency figures (Clef faster than Jev) reversed from Japan, round trip included. `unverified`, one author; n small (8 items for repeats) |
| Japanese decision set v1, 260 items frozen and hashed (support routing 60, agent actions 80, risk review 60, model routing 60; colloquial, typos, Japanese-English mix) ([jev-vs-openai-dicision-api](https://github.com/mantaroh/jev-vs-openai-dicision-api), [@_mantaroh_](https://x.com/_mantaroh_/status/2106713830885855426), run 2026-10-04, concurrency 4) | Jev vs Clef, Clef-flash | Accuracy 92.69 / 91.15 / 92.31%; macro-F1 92.30 / 90.33 / 92.24; ECE **0.074** / 0.202 / 0.293; Brier 0.121 / 0.141 / 0.144; p50 to response headers 203 / 452 / 249 ms (p99 341 / 1,482 / 733); $0.0194 / $0.0526 / $0.0197 per 1,000 requests | Calibration, latency; accuracy and (with Clef-flash) cost about tied | Jev's tokenizer counted ~2.1× the input tokens (120,066 vs 56,934), so compare cost per request, not per token (author). Jev's $0.0050 total matches $0.042/M input: `verified` (our arithmetic, [[reference/models-and-pricing]]). One run; repo licence not stated; `unverified` |
| An internal agent's own tasks, 224 calls per model, 2026-10-02 to 03 ([@glebedel](https://x.com/glebedel/status/2106795473965097291), replying to a viral claim that Clef outperforms Jev; chart transcribed, latencies read off the chart) | Jev vs Clef, Clef-flash | Yes/no and routing (146 cases) 99 / 95 / 95%; files a task needs (40) 75 / **100** / 83%; memories to load (16 tasks, F1) 80 / 77 / 65; preference clashes (103 pairs, F1) 97 / 97 / 74. Average time per call ~0.5-0.8 s (Jev) vs ~0.6-1.35 s (Clef), network included; Clef-flash fastest on 3 of 4 | Jev or tie on 3 of 4; **LOST** file selection to Clef | `unverified`: internal tasks, labels and method not published; 16-146 items per task |
| Own app tasks via OpenRouter, earlier models run 2026-09-25 and 28 ([@iliaa](https://x.com/iliaa/status/2106780379890016705), 2026-10-04; table transcribed) | Jev vs Clef, Clef Flash, Kev 4B, Solar Decide, Span-01 (none of the five is Jev) | Jev / Clef / Clef Flash / Kev 4B / Solar Decide / Span-01: review 16 / 16 / 15 / **18** / 15 / 12 of 24; false merges 5 / 6 / 4 / 5 / 0 / 1 of 16; duplicates found 8 / 8 / 7 / 8 / 0 / 3 of 8; skill at 0.5 **63** / 61 / 60 / 61 / 51 / 52 of 64; support booleans 103 / **112** / 110 / 111 / 110 / 102 of 120; intent **39** / 38 / 37 / 34 / 35 / 28 of 40; review median **166** / 665 / 375 / 540 / 929 / 393 ms; cost $0.00792 / $0.04125 / $0.01547 / $0.00622 / $0.01273 / $0.00548. Jev's 2026-09-25 run: same except false merges 4, 221 ms | Jev on skill, intent, latency (author: "still well in the lead"); **LOST** support booleans to Clef, Kev and Solar | `unverified`: tasks and labels not published, small counts, one run each. Span-01's asterisk is not explained. Kev: [[ideas/open-replicas]] |
| One support state ("checkout failing for every customer for an hour") with Cloudflare's sample `noul`, 3-option `choice` and 4-level `score`, 5 runs each ([DevelopersIO](https://dev.classmethod.jp/en/articles/cloudflare-clef-overview/), Kema, Classmethod, 2026-10-04) | Jev (`jev-1.13.0`) vs Clef, Clef-flash | Same decisions from all three (urgent, `technical`, severity near Critical). Median 238 / 478 / 294 ms; $0.0000169 / $0.0000830 / $0.0000311 per request (402 vs 346 input tokens). Jev's probabilities came with two decimals and moved in the second across runs (urgent 0.96-0.97); Clef's four-decimal values were identical across all 5 | Jev on latency and cost | n = 5 on one state; the author says timing varies and this is too few runs for a speed ranking. Jev's cost matches $0.042/M: `verified` (our arithmetic). Over the REST API Clef wraps answers in `result`: [[ideas/rival-decision-models]] |
| Translation workflow decisions: term starts and Khmer word choice, 1,248 test states per Cloudflare model; latency on 100 matched live cases ([@useLectio](https://x.com/useLectio/status/2106042999516139859), Lectio Lab card 2026-10-01; transcribed) | Jev 1.13.0 vs Clef, Clef-flash | Term starts, Khmer: 19/22 vs 1/22 vs 0/22; eight other languages 43/53 vs 29/53 vs 6/53. Khmer word-choice errors found (meaning notes supplied) 11/17 vs 0/17 vs 0/17; false alerts 0/30 for all three. Median latency 131 / 772 / 680 ms | Jev; they kept it | Small cohorts scored against saved references, score cutoff ≥ 0.90 (the card: not a universal ranking); `unverified` |
| Gateway request log, a few calls each ([@leighchalliday](https://x.com/leighchalliday/status/2106492087428427829), 2026-10-03; screenshot transcribed) | `typesafe/jev` vs `@cf/cloudflare/clef` | Jev: 8 calls, 128-170 ms, ~$0.00008-0.00024 each (1.9k-5.6k input tokens); Clef: 7 calls, 624-1,156 ms, $0.00059 each for 6 of them | Jev (~5× faster, cheaper per call) | Anecdote: calls on different days, inputs likely differ, gateway not named. Jev's per-call costs match $0.042 per M input tokens with output free: `verified` (our arithmetic, [[reference/models-and-pricing]]) |
| Cost on identical synthetic text, 1 to 64 questions over shared state ([@xadhish](https://x.com/xadhish/status/2106085251995115618), 2026-10-02; [raw outputs and code](https://decision-api-cost-study.adhishthite.chatgpt.site), not captured) | Jev vs Perplexity, Clef, Clef Flash | With 64 questions Perplexity cost 26.3× Jev; with one question Perplexity was cheaper | Cost at many questions; **LOST** at one | `unverified`: post only; cost only, no accuracy |
| s1bench, 1,999 items from six public subsets (VitaminC dev 599, MASSIVE en-US 350, BoolQ 300, HelpSteer2, Aegis2, PAWS 250 each), same prompts for every model ([bench.jakecuth.com](https://bench.jakecuth.com/), [@ItsCuthulhu](https://x.com/ItsCuthulhu/status/2107218156330614945), 2026-10-05; board data captured 2026-10-06) | Jev (API) vs Clef-flash (9B) and Clef (27B), open weights on the author's own hardware, and Decision-2.0-Nox-4B (vllm-sr, CPU); about 50 open models and replicas on the same board then, 109 entries by 2026-10-14 (re-read: Jev's run and these comparisons unchanged; standing on [[ideas/eval-boards]]) | Macro accuracy **0.775** vs Clef-flash 0.757 (items only Jev got right 175, only Clef-flash 136; p = 0.031), Clef 0.747 (p = 0.0009), Nox-4B 0.742. Jev / Clef-flash per subset: VitaminC 0.803 / 0.766, MASSIVE 0.874 / **0.906**, BoolQ 0.893 / 0.860, HelpSteer2 0.348 / **0.380**, Aegis2 0.836 / 0.820, PAWS **0.896** / 0.808. ECE 0.076 vs **0.043** (Clef 0.062). Median 0.42 s (API) vs 0.12 s (Clef-flash), 0.38 s (Clef), 4.7 s (Nox-4B); 0 errors | Accuracy; **LOST** calibration and latency to Clef-flash | One live board (it refreshes; numbers as of our capture). Clef timed locally, Jev over the network. The board's own Δ column compares with "published Jev 1.13.0 numbers" (source not named; macro 0.768), not with this live run. Replicas above Jev on the same board: [[ideas/open-replicas]]. `unverified` |
| Agent-harness reflex layer: rate each tool call reading / undoable / can't undo; 1,000 logged calls from 18 days replayed (916 shell commands, excerpts of ≤ 80 characters, sample balanced across Jev's three logged answers) ([@daveremy](https://x.com/daveremy/status/2107194467723997200) and its X article, 2026-10-05) | Jev vs Clef; disagreements graded blind by Claude Opus 5.5 and GPT-6 Astra | Median 115 vs 400 ms (p95 195 vs 826; Jev faster on 699 of 700 timed calls). Same level on 811 of 1,000. On 50 of 177 distinct disagreements: grader 1 sided with Clef 33 vs 11 (of 44), 23 vs 21 after relabelling 12 status-record calls; grader 2 Jev 17 vs Clef 15 (of 32); the graders agreed on 29 of 50 (kappa 0.41). Estimated $0.018 vs $0.096 per 1,000 decisions (input tokens at list price, not a bill) | Latency, cost; accuracy unresolved (author kept Jev) | Only disagreements graded, by AI graders whose ranking moved with the labels; short excerpts (in production Jev sees up to 4,000 characters); one machine, 87% of calls from one agent. Jev's estimate implies ~430 input tokens per call at $0.042/M, plausible for a short excerpt plus the ~260-token request overhead ([[ideas/request-mechanics]]; our arithmetic). `unverified` |
| Policy positions in political-debate transcripts: does a quote state a real position? 14 sampled items scored by hand; also Banking77 and 60 fresh questions ([@samueljmcd](https://x.com/samueljmcd/status/2107077637617180682), [method reply](https://x.com/samueljmcd/status/2107132236864643230), 2026-10-05) | Jev vs Clef | Own task: Jev 13 of 14, Clef 9 of 14. Banking77: Clef ahead by 19 points; 60 fresh questions: tie (scores not given) | Own task; **LOST** Banking77 | n = 14; Banking77 sample size and method not stated. The author's view that Clef reproduced the dataset's odd labels and was "very likely trained on it" is their inference: `unverified`, no training-data source checked. `unverified` |
| Own benchmark, fresh and uncached ([@MKhordoo](https://x.com/MKhordoo/status/2106157368694522060), 2026-10-02, a reply in a thread we did not capture) | GLiDE vs "JEV" | Accuracy 89.7% vs 33.2%. The post's latency figures (median 6.8 → 0.81 s, mean 4.56 s, p95 12.9 s when ~325 thinking tokens kick in) describe a system with a reasoning path; Jev generates no tokens, so they are not Jev's (inferred; which system they describe is unclear) | **LOST** (as posted) | `unverified`: task, n and what "JEV" names (TypeSafe's API or another service) not stated; 33.2% is far below every other run on this page. Thinking tokens read as Jev's would `contradict docs` ([[concepts/system-one]]) |
| Production decisions, ~5,900 in all: email triage, document tasks, financial line items mapped to standard categories ([@elifoltyn](https://x.com/elifoltyn/status/2107529791729496402), 2026-10-06) | Jev vs Perplexity's Decider V1 27B (pplx-decider) | Email triage 84.4 vs 83.6%; documents 83 vs 75%; line-item mapping 74 vs 58%; same speed; Decider 2-11× the cost | Jev | `unverified`: post only; per-task counts, labels and method not stated. The author puts the cost gap down to Jev billing per call and Decider per decision; Perplexity lists a per-input-token price ([[ideas/rival-decision-models]]), so the gap more likely comes from how input is counted across questions (inferred; @xadhish's many-question cost run above shows the same shape) |
| Claim verification over SEC 10-K excerpts (400) and the 12-subtask LegalBench slice (396), the same sets, prompts, scoring and statistics as Vals' Jev evaluation ([[ideas/head-to-head-benchmarks]]); latency for 1 to 64 judgments about one long document ([Vals AI](https://www.vals.ai/blogs/evaluation-of-mercury-decide), Connor Frank, 2026-10-08; independent evaluator) | Jev 1.13.0 vs Inception's Mercury Decide (one call per item through Inception's API, no retries, no version pin) | Claims: 0.978 vs Jev 0.975, paired difference −0.017 to +0.021 (indistinguishable); subtle contradictions 0.988 vs 0.912, direct support 0.963 vs 1.000. LegalBench: **0.869** vs 0.740 accuracy, class-balanced 0.821 vs 0.730; most of the gap from `contract_nli_permissible_copy` and `contract_nli_confidentiality_of_agreement` (0.970 on both vs Jev 0.485 and 0.576); Mercury's weakest subtask `international_citizenship_questions` 0.70 vs Jev 0.82. ECE 0.011 for both on claims, **0.058** vs 0.108 on LegalBench. 1% error budget (cut set on 30% of claims, tested on 265 held out): Mercury 94% coverage at **0.8%** error (within), Jev 95% at 1.6% (over). Latency: Mercury median 0.16 s at 1 judgment, 0.40 s at 64; Jev stays near 0.1 s; on the same claims at the same moment Mercury's median was 1.44× Jev's. Per 1,000 claims $0.006 vs $0.025 (lowest of 13 systems). Mercury's billed input grows from ~1,700 to ~106,000 tokens from 1 to 64 judgments: $0.0022 per 64-judgment call vs $0.00016 for Jev | Latency; cost with many questions per document; claims tied. **LOST** LegalBench, LegalBench calibration, the error budget and cost per single-question claim to Mercury | Vals priced Mercury at its "$0.021" launch price; Inception's docs say $0.02, 50% off a $0.04 list ([[ideas/rival-decision-models]]). No version pin, and Inception shipped an update the same day (2026-10-08), so which version Vals measured is not stated (inferred). LegalBench slice small and public (33 items per subtask). The cost crossover, our arithmetic: [[ideas/cost-ledger]] |
| Answer verification: 280 labelled LLM-AggreFact answers, each checked against its source with several requests, split into five input-size groups of 56 (~600 to ~6,100 characters); four in flight, caches off, 2026-10-07 and 08 ([CalibratedAgents](https://calibratedagents.com/blog/decision-apis-benchmark-2026), [@calibratedagent](https://x.com/calibratedagent/status/2108199862005313694); **vendor**: sells an answer-checking API) | Jev vs OpenAI's Decisions API (`gpt-6-luna`) vs Clef-flash | Median seconds per answer, smallest → largest group: Jev 0.95 → 1.04 (p90 1.1 → 1.4); OpenAI 0.46 → 2.7 (p90 7); Clef-flash 1.7 → 2.4. Per 1,000 answers at list price: Jev $0.47 → $0.71, OpenAI $1.18 → $3.12, Clef-flash $1.15-$2.06; on the largest group Jev counted 16,900 tokens per answer, OpenAI 31,200. Balanced accuracy as a one-question judge: OpenAI 76.8, Jev 76.3, Clef-flash 68.2. Best thresholds on development answers: Jev 0.25, OpenAI 0.11, Clef-flash 0.28 | Latency on large inputs, cost; accuracy tied with OpenAI. **LOST** latency on small inputs to OpenAI | The FAQ's lower bounds ($0.43 Jev, $1.13 OpenAI, 0.45 s) differ from the body's ($0.47, $1.18, 0.46 s). Clef-flash's ~$0.09 price is from launch coverage; its token count fell on the two largest groups (cause unconfirmed). Cached first runs looked 25% faster and half the cost; OpenAI's rate limit was hit at four in flight. Jev's largest-group cost matches 16,900 × $0.042/M = $0.00071: `verified` (our arithmetic, [[reference/models-and-pricing]]). Otherwise `unverified` |
| Dictation: should a word the personal dictionary flags be swapped? 121 flagged places in 2.1 h (160 recordings) held out from the 7.5 h the dictionary learned from; answers written down before any run; one speaker, one reviewer, one run, in the open-source Entune ([@eandualem](https://x.com/eandualem/status/2108157205883257084), 2026-10-08; card transcribed) | Jev vs Perplexity pplx-decider (open weights), OpenAI's Decisions API, Laya (local on a Mac; not Jev, [[ideas/open-replicas-laya]]) | Fixes caught of 47 / wrong swaps of 72: Perplexity 43 / 3; Jev 38 / **1**; OpenAI 42 / 8; Laya 28 / 8 (always swapping: 47 / 72) | Fewest wrong swaps; **LOST** fixes caught to Perplexity and OpenAI | 47 + 72 = 119, not the 121 stated. One speaker, small n; pplx-decider build and thresholds not stated. `unverified` |
| Email security, 1,985 non-public synthetic emails (785 attacks), is_attack + 4-way verdict + 5-way attack type in one request, 80% test split ([X article](https://x.com/ShrivuShankar/status/2108218472476168409), Abnormal; [@ShrivuShankar](https://x.com/ShrivuShankar/status/2108219279820931075), 2026-10-08; charts transcribed) | jev-1.13 vs OpenAI Decisions (luna-6 decisions), 34 self-hosted open decision models, Sonnet 5.5 and Luna-6 low/max; 40 models | is_attack ECE Jev **0.012**, best of all (kev-9b 0.014, Decisions 0.023, luna-6-max 0.028, sonnet-5.5-low 0.059). Average precision Jev 87.7, Decisions 93.3, sonnet-5.5-low 97.6. Jev and Decisions on the cost/email-score Pareto frontier (chart reading: Jev about 62 at about $0.09/1k emails, Decisions about 70 at about $0.20). Recall at 99% precision: Jev about 10% (frontier on price only), Decisions about 38%, luna-6-max about 45%; a $200 LoRA of kev-4b 7% -> 66% | Calibration; **LOST** recall at high precision | Non-public synthetic set; thresholds picked on the test split; costs are estimates; precision computed at a 50% attack rate; production detection costs about a cent per 1,000 emails, so decision models only behind prefilters (author) |
| Answering model inside a decision harness (think-reason-learn, Vela Research: LLM-built yes/no rules and policies scored per case); 12 medicine, law and finance benchmarks, 53,057 cases per full run ([@yihlamur](https://x.com/yihlamur/status/2108586553429926185), 2026-10-09; card transcribed) | Jev vs OpenAI GPT-6 Luna Decisions, Cloudflare Clef Flash, Gemini 2.5 Flash | Ranking accuracy (how often a success ranks above a failure), Random Rule Forest / Policy Induction: Jev 68.6 / 69.5%, Luna ≈69.0 / ≈71.5% (sampled), Clef Flash 67.0 / 66.8, Gemini 63.9 / 68.3. Run cost $5.77 / ≈$27 / $4.16 / ≈$74 (frontier models: $350-900, author). Luna declined 6% of cases, unevenly (court rulings, reviews of real people) | Cost vs Luna and Gemini, kept Jev; **LOST** accuracy narrowly to Luna, cost to Clef Flash (28% cheaper, slightly worse) | `unverified`: card and post only; Luna's accuracy sampled, not a full run; per-benchmark numbers not given. The public [think-reason-learn](https://github.com/Vela-Research/think-reason-learn) repo had no Jev backend at our 2026-10-10 capture. Refusals make Luna's scores not comparable with models that answer every case (author) |
| Thirteen decision models through OpenRouter's Decisions API, one question per request, 633 items: AG News (4 labels) and 27 support tickets; MASSIVE (59) and Banking77 (77); WikiToxic and Amazon polarity as yes/no, fixed-seed BTZSC samples ([jev-measured](https://github.com/WallerChen/jev-measured) §8, 2026-10-08; linked from a [dev.to how-to](https://dev.to/weirui_chen_48669a172a6be/typesafejev-fails-on-openrouter-chatcompletions-how-to-call-it-and-the-three-response-shapes-4e0f) by its author, who runs jev-agent.com, not TypeSafe) | `typesafe/jev-1.13` vs d1, Decider V1.1 27B, Mercury Decide, Clef, Kev 4B, Solar Decide and Flash, GPT-6 Luna Decisions, Clef Flash, Tev1 4B, Span-01 and Lite (none is Jev) | Jev simple choices 94.0%, 59-77 options **79.0%**, yes/no 95.0%; best: d1 96.5 / Clef Flash 92.0 / Clef and Clef Flash 97.0. Wrong at ≥ 90% confidence: Jev **11.5%**, Clef 2.3, d1 3.2, Luna Decisions 9.2. p50 184 ms (fastest 128-138: Span-01, Tev1, Luna Decisions); $0.0304 per 1,000 decisions (Luna Decisions $0.0401, d1 $0.0078, Clef $0.1541). Solar Decide (≤ 26 options) and Tev1 (≤ 20) refused the large sets | **LOST** accuracy and confident-error rate to several; mid-pack on cost and latency | One run, from Vercel `iad1`, all through one gateway; 100 items per set. At list price Jev's cost ≈ 720 input tokens per decision (our arithmetic). Laya on the same items is §9 of the same repo (not Jev; [[ideas/open-replicas-laya]]) |
| Support routing to 4 teams (billing, technical, sales, other), 16 Japanese inquiries the author wrote (4 per team, 4 where "other" is right), 3 passes one at a time, 2026-10-05 ([BugattiAlpha blog](https://bugattialpha.com/trends/ollama-clef-flash-decision-model/), [post](https://x.com/BugattiAlpha/status/2109252747078119738), 2026-10-11; Japanese) | Jev (`jev-latest`, API) vs Clef-Flash 9B (Q8_0) on Ollama 0.35.1 in WSL2, an 8 GB RTX 2070 SUPER (41% CPU / 59% GPU) | Both 16/16, all 4 "other" right. Median 233 ms (network included) vs 739 ms (model loaded); medians over 3 passes 233-243 vs 735-739 ms; 16 calls 3.89 vs 12.37 s. `confidence` 0.80-1.00 vs 0.38-0.94; Jev's moved on 2 items across passes (at most 0.06), Clef-Flash's identical to 3 decimals. Input tokens 7,091 vs 3,359; Jev about $0.0003 | Latency; accuracy tie | 16 easy author-written items, one PC. Jev's `confidence` fit the documented Choice formula (`verified`, [[concepts/confidence]]); Clef-Flash's fit 1 − H/ln n on all 16 (author, inferred from the values, not from Cloudflare's docs), so it runs lower on the same right answers and a Jev cut such as 0.9 would send most of them to review (author). Windows Ollama 0.35.1 failed every request with "Clef: non-finite logit", CPU-only too, as in open Ollama issue #18769; the Linux build in WSL2 ran the same model file. Cost: 7,091 × $0.042/M = $0.0003 (`verified`, [[reference/models-and-pricing]]) |
| Tool-call safety gate for a home assistant and Claude Code: three Nouls (deletes something? within the request? touches secrets?) on about 1,000 calls a day, the author's own logs ([note.com article](https://note.com/aynyan_sui_ice/n/n016e9552cfa9), [@Aynyann](https://x.com/Aynyann/status/2109016956968096236), 2026-10-10; Japanese) | Jev in production vs Kev (open, **not Jev**: [[ideas/open-replicas]]) on a Mac with its last layer retrained on the logs (70 min), shadowing 11,281 operations 2026-09-26 to 10-07; vs Cloudflare Clef and Clef-flash on 400 operations from 2026-10-03 to 07, about half of them ones Jev leaned risky on | Kev's "deletes?" before retraining: 0.39 on safe vs 0.45 on risky operations; after, AUC 0.77 → 0.995. Shadow run: Kev passed 0 operations Jev leaned risky on (the switch condition), but passed 543 without asking vs Jev's 1,434 and blocked 0 vs Jev's 33. Clef passed 80 that Jev sent for confirmation (39 Jev read as deleting), Clef-flash 35 (19): `sed -i`, Python and `cp` overwrites, `ffmpeg -y`, a config.json rewrite; neither blocked any of Jev's 4 blocks. From Japan both about 0.6 s. Jev cost $0.31 for September, $0.39 for October so far | Jev kept (it stops things) | Yardstick is agreement with Jev, not labels: whether Clef's passes were wrong depends on the author's rule "never overwrite config"; Jev also blocked an in-scope `sed` edit (author). Kev stays as the offline fallback. At ~900k tokens a day Clef-flash fits Cloudflare's free tier, Clef ~$3 a month (author's estimate) |
| Japanese RAG answerability gate on JQaRA (quiz questions with Wikipedia passages): A, one passage, "can this answer it?" (1,000); B, five passages, "is the answer among them?" (500); C, eight answer-preserving rewrites plus two identical resends (150); one instruction for all, cut at 0.5; run 2026-10-10, all decision models via OpenRouter ([@tbs_noguchi](https://x.com/tbs_noguchi/status/2108914269039517783), [note.com write-up](https://note.com/tbs_noguchi/n/n60777d9b429f), Japanese) | Jev (`typesafe/jev-1.13`) vs Microsoft-Decision-1 (-20261009), GPT-6 Luna Decisions; GPT-6 Luna and Claude Haiku 5.5, with and without reasoning, as reference | Accuracy A / B: Jev **0.724** / **0.816**, MD-1 0.672 / 0.728, Luna Decisions 0.680 / 0.770 (the author: Jev's lead is beyond chance). Median latency 0.40 / 0.46 / 0.42 s (Jev / MD-1 / Luna Decisions; ~0.16 s of it network); reference LLMs 0.9-1.4 s. MD-1 cheapest (about JPY 2 per 1,000); Luna Decisions dearest (JPY 8.1 / 19.3 per 1,000 on A / B); MD-1 and Jev share a list price but Jev counted 2.6× the tokens on the same Japanese input. False "answerable" 0.6-1.0% (A) and 4.4-6.0% (B) for all three; most errors are misses; MD-1 put nearly half the items under 1%, so the 0.5 cut lost answerable ones. Flips over the 8 rewrites: Jev 2.5% on average (each ≤ 3.3%), MD-1 3.7% (Microsoft publishes 1.3%; reversed option order 0.0% as published), Luna Decisions 5.8% but identical probabilities on identical resends; MD-1 and Luna 10% when passages or fields were reordered; Haiku 15.3% on a reworded instruction. MD-1 moved 0.44 → 0.62 → 0.53 on identical resends; Jev moved at most 0.02 against the author's earlier run, but up to 0.11 when one option-description line was added. GPT-6 Luna with medium reasoning was the most accurate setting (no probabilities) | Jev on accuracy and stability; MD-1 on cost; Luna Decisions on repeat determinism | One run, one public dataset (possibly in training data); 27% of answer-containing A items were rejected by all seven settings, a label-vs-question mismatch the author did not fully hand-check; Microsoft's variations and data are not public, so conditions differ; MD-1's version cannot be pinned; MD-1 calls landed on Azure. `unverified` |
| Agent decisions inside hedwigai ask (an MCP research tool): 92 yes/no questions on public companies (biotech above a size; traps: made-up companies, mismatched tickers, wrong exchange; close calls within 6%), answers checked against SEC records; ~81 decisions per question in ~2.2 requests; same build, same day, 2026-10-10 ([@aloobhujiyan](https://x.com/aloobhujiyan/status/2108963094026486168), [hedwigai blog](https://www.hedwigai.com/blog/decider-in-ask); vendor: hedwigai) | ask with Jev vs ask with pplx-decider-v1-27b, self-hosted (8-bit, two RTX 3090s, 16 decisions at a time) | Right 89 / 89, wrong 1 / 2, unsure 2 / 1 (Jev / Decider); right of answered 99% / 98%; median per question 2.1 s vs 19.1 s (the post blames the two cards). With one lookup described as "NOT a market cap", Decider picked "none of these" on 59 of 92 and left 23 unsure (67 right); reworded as the closest stand-in, it got 89 right; Jev's answers did not change under either wording | Tie on accuracy (the post says "jev still is better"); Jev on latency and on robustness to wording | One run per model, one question type, 92 questions; Decider on consumer GPUs, not Perplexity's API; Jev billed per call, Decider has no per-decision fee. `unverified` |
| 200 random comments from Better Stack's YouTube channel, five questions (spam, worth replying to, says the video is wrong: Nouls; tone: 5-option Choice; toxicity: Score); Claude Opus 5.5 answered the same questions as referee, a match = the referee's top answer at ≥ 50% ([Better Stack guide](https://betterstack.com/community/guides/ai/jev-vs-clef-decision-model/), Stanley Ulili, updated 2026-10-12, with a video) | Jev vs Clef (27B, Workers AI) | Overall 86 vs 83% (the author: "roughly equivalent"). Spam 98 / 96; worth replying 81 / **86**; says the video is wrong 96 / **97**; tone **75** / 49 (Clef leaned to "neutral"); toxicity 81 / **88**. Jev "nearly twice as quickly" and 6.5× cheaper per comment, though the whole Clef run fitted in Workers AI's free daily allowance. A 225-thumbnail test ran on Clef only (Jev reads text only) | Overall, through tone; **LOST** 3 of 5 questions | Agreement with one referee model, not ground truth; one run. The guide says a Score returns a float between 0 and 1: `contradicts docs` (`score` runs from 0 to the number of levels minus 1, [[concepts/score]]). Text only, $0.042 per M input, free output, 32K: `verified` ([[reference/models-and-pricing]]) |
| Position bias as a judge: all 1,865 RewardBench 2 prompts, four responses rotated through every slot (76,200 requests per model, 381,000 in all), preregistered ±3 pp margin; a second arm keeps the responses fixed and rotates the option or question order ([order-blind-decisions](https://github.com/dorkitude/order-blind-decisions), Kyle Wild, [post](https://dorkitude.com/writing/order-blind-decisions), [@dorkitude](https://x.com/dorkitude/status/2109955038147989693), 2026-10-13) | Jev (`jev-1.13.0`) vs OpenAI Decisions (`gpt-6-luna`), Clef-flash, Microsoft-Decision-1 (via OpenRouter), SubQ Snap (preview) | Choice accuracy **80.9%** / 78.2 / 62.7 / 70.9 / 78.5. Right answer picked, slot 1 minus the middle slots: Jev +0.2 pp [−0.8, +1.2] (order-blind under the frozen rule), Decisions +6.1, SubQ +7.0 (both outside the margin); Microsoft −4.5 pp at the last slot; Clef-flash inconclusive. Same winner in all four orderings: Jev **79.2%**, others 64.5-78.2%. Exact ties (two equally right answers), earlier one picked: 71.5 / 88.6 / 69.1 / 52.8 / 68.1% (50% is fair). Option order only: Jev's pick changed in 6.0% (Clef-flash 2.5, Microsoft 5.1, SubQ 14.0, Decisions 21.8); Jev's first-option lift +0.6 pp [+0.1, +1.0]. Run cost $6.94 / 14.76 / 14.86 / 5.90 / 7.35; p50 105 / 113 / 260 / 307 / 274 ms. Decisions refused 1,786 requests (mostly the Safety subset); the others none | Jev on accuracy, position bias and latency | One study, one author, code and frozen requests public. Jev's small lean to the first option on option order and on exact ties fits documented failure mode 8 ([[concepts/jaggedness-jev-1-13]]): consistent with docs, not a contradiction, and small here. The author reads the rivals as built on fast LLMs (inferred, most do not publish an architecture) |
| Long logs: 3,840 questions about a made-up store's event log (find, cross-check, count; a test re-derives every answer from the log text), 2K to 360K tokens, 20 logs per size, byte-identical requests in alternating order, run 2026-10-13 from Spain ([subq-vs-jev](https://github.com/jcarlosrodicio/subq-vs-jev), [results page](https://jcarlosrodicio.github.io/subq-vs-jev), [@JuancaRodicio](https://x.com/JuancaRodicio/status/2110028462186451315); independent, SubQ access free during its preview) | Jev vs Subquadratic's SubQ Snap (preview, 384K window; not Jev) | Logs up to 30K tokens: SubQ 93.9 vs Jev **94.7%** (the post: SubQ 95% at 16K). 360K logs: SubQ 58.8% in one call vs Jev **77.1%** reading 29K chunks, answers combined. ECE up to 30K 0.028 vs 0.038; 64K and up 0.163 vs 0.117. Byte-identical output on repeat 90.0 vs 50.8%. Shuffled options changed 3 vs 5 of 80 answers. Median at 8K 0.75 vs **0.31 s** | Long-context accuracy (chunked), latency; **LOST** repeatability, short-context calibration | Jev rejected requests past about 32.7K tokens with `max_tokens_exceeded`, so chunking was the author's code: fits the documented 32k budget, `verified` ([[reference/models-and-pricing]]; the error code is undocumented, [[reference/rate-limits-and-errors]]). The README says Jev's output price was unpublished: `contradicts docs` (output tokens are free). SubQ changed between the 10-10 and 10-13 runs under the same model name (11% of answers byte-identical; up-to-30K accuracy 96.7 → 94.4%) |
| Mixed small bench: 27 cases × 3 repeats over images, files, text and guardrail checks, 2026-10-12 ([@GazvodaMatjaz](https://x.com/GazvodaMatjaz/status/2109720500154851367); card transcribed) | Jev vs SubQ Snap | Calls succeeded 54/81 vs 81/81 (Jev returned HTTP 400 on all 27 image cases); accuracy on answered questions 100% (108) vs 100% (153); median 252 vs 299 ms, p95 353 vs 401; input tokens per call 392 vs 219; mean confidence 0.931 vs 0.940 | Latency; **LOST** coverage (images) | Tiny set. Jev takes text only, so the image failures are expected: `verified` ([[reference/models-and-pricing]]); the card calls them a request-format issue. Also seen, no numbers: [@Numankhannnnn](https://x.com/Numankhannnnn/status/2109685068256166087) (600 questions: tie on accuracy and speed; SubQ takes ~11× more text and reads images) and [@VladTerin](https://x.com/VladTerin/status/2109790097767891325) (browser use: SubQ a little slower and less accurate, "very comparable") |
| Predicting people: the share of players choosing action A in 2,408 matched two-player game situations (Games2p2k, Zhu et al. 2025) ([@mblukac](https://x.com/mblukac/status/2111087060886888560), 2026-10-16, [metric reply](https://x.com/mblukac/status/2111115710759849990); an earlier Jev-only run on 2,416 situations, [@mblukac](https://x.com/mblukac/status/2110672816475418755), 2026-10-15; charts transcribed) | Jev (1.13.0 in the earlier chart) vs Liquid d1-ultra (not Jev); OpenAI Decisions also plotted | Average error 10.9 pp (d1-ultra) vs 11.8 pp (Jev); "Jev wins on calibration", measured by Brier score and a 10-bin calibration error (values not published). The earlier Jev run cost about six cents. In the plot, OpenAI Decisions predicts A too often (our visual reading) | **LOST** average error, narrowly; calibration claimed for Jev | `unverified` calibration: no values given; one run each. Predicting human behaviour is outside what Jev was built for; the author probes it on purpose |
| Decision models vs cheap LLMs on the author's own tasks: claim checks (including adding up claim totals), routing, Banking77, sentiment ([@drinkzima](https://x.com/drinkzima/status/2112250440893301197), Colin Zima, founder of Omni, 2026-10-19) | Jev vs Decision-1 (Microsoft-Decision-1, inferred), Laya, Clef (none is Jev), OpenAI Decisions (GPT-6 Luna); GPT-6 Luna and Claude Haiku 5.5 as chat models | Jev's only number: its stated probability averaged about 89% where it was right 93% (underconfident; "averaged" inferred); Decision-1 84% vs 97% right; Laya and Clef ~65% vs 97%. Comparator context, no Jev figure given: Decision-1 99% on claim totals vs 48-55% "for others" (whether Jev is among them is not stated), 99.5% on all claim checks; Luna 99.5% on claim checks as a chat model vs 68.5% through Decisions, Banking77 89.6 vs 78.3%, Decisions 6.09¢ vs 1.30¢ per 1,000 rows; at temperature 0 Luna and Haiku kept 85.1% and 88.1% of routing picks on identical reruns; Laya answered "moderate" on 1,532 of 1,534 routing questions, 96.6% on sentiment | Not stated for Jev; Jev underconfident, not overconfident | `unverified`: post only; task sizes, labels and method not published; it links omni.co, where we found no write-up (2026-10-20). Author: test the actual task. Adding up is arithmetic, which Jev's docs say to keep in code ([[concepts/jaggedness-jev-1-13]] mode 2). Decision-1 vendor facts: [[ideas/rival-decision-models]] |
| Chinese content moderation: 825 samples from 5 public sets (COLD, ToxiCN, spam SMS, Safety-Prompts, NSFW image captions; 63.6% violations, gold mapped from each set's labels), 4 questions per request (8-way Choice, severity Score, 2 Nouls) plus fixed thresholds, concurrency 4, 2026-10-23 and 24 ([REPORT.md](https://github.com/MuShan-bit/decision-moderation-bench/blob/7d843be8414e96d9d43a617e7c063dc74dfc0b94/REPORT.md), [@mushan40120676](https://x.com/mushan40120676/status/2113829740838797665)) | `jev-preview` (`jev-1.14.0` on 2026-10-24) vs StartLux-Decision 27B (via BambooNode) vs Alibaba Bailian's `decision-model-preview` (DashScope) | Accuracy 79.9 / 83.8 / 73.2%; F1 0.851 / 0.874 / 0.774; recall 90.3 / 88.2 / 72.2%; false positives 115 / 72 / 75. End-to-end p50 313 ms / 1,533 ms / 130 ms; p95 359 ms / 7.98 s / 1.94 s; 12.46 / 1.58 / 11.03 req/s. Input tokens per sample 1,141 / 848 / 573. Bailian's wrong answers had median confidence 0.94 | **LOST** accuracy to StartLux; best recall, p95 and throughput; **LOST** p50 to Bailian | Author's own run from China through a local proxy (~5% of Bailian requests stalled 1-10 s); gold labels not re-checked. Same request, about twice Bailian's token count: different tokenizers (inferred). StartLux and Bailian are not Jev ([[ideas/rival-decision-models]]) |
| Decisions in an applicant tracking system (ATS), shipped in the author's product ([@crypto_guro](https://x.com/crypto_guro/status/2112570525721759966), 2026-10-20) | Jev vs GLiDE | "Nearly identical results" (no numbers); picked Jev for speed and cost | Speed, cost; accuracy tie | `unverified`: one post, no task detail, n or figures |

Microsoft-Decision-1 vendor numbers (36 benchmarks, Jev included): [[ideas/rival-decision-models]]. SubQ Snap vs Jev on the s1bench board and in chess: [[ideas/eval-boards]]. Jev 1.14 images vs YOLO, SCRFD, ArcFace and cloud vision APIs on home-camera frames: [[ideas/builds-games-and-devices]]. Ad-appeal scoring of 56 videos, Jev on extracted words vs Qwen and Luna Decisions on frames plus words (Reflections, @Sumanth_n83): [[ideas/builds-content-and-media]].

Pac-Man benchmark across Jev, Clef, Clef Flash, GPT-6 Luna Decisions, Kev 4B and Laya (Opper's jevman): [[ideas/head-to-head-games]].

Vega (open, not Jev) vs Jev 1.13.0 on author-selected tasks, incl. OCR-then-Jev on document images: [[ideas/open-replicas-laya]] (Vega). Fine-tuned Laya vs zero-shot Jev on toxic comments: [[ideas/open-replicas-laya]] (John Mark).

Also, no numbers: [@heyitsnoah](https://x.com/heyitsnoah/status/2106748138027372964) (2026-10-04) tried Clef in a few places they run Jev in production and found it "both worse and significantly slower"; they call the sample small.

Japanese decisions, coverage at 95% accuracy, Jev vs Liquid d1, Cloudflare Clef/Clef-flash, Kev-4B, Claude Haiku 4.5, with latency and cost (Gota, Findy deck): [[ideas/builder-lessons]].

One production week repriced at Clef and Clef-flash list prices (Sorokin, 2026-10-05): [[ideas/cost-ledger]].

James Briggs's Jev vs GLiNER2.5, GLiGuard, Von 1.1 and Laya (Banking77, CLINC150, BFCL, guardrail sets, calibration, latency): [[ideas/rival-decision-models]] (GLiNER2.5 row).

## Related

- [[ideas/rival-decision-models]] — the rival models themselves and their vendors' numbers
- [[ideas/head-to-head-decisions-api]] — Jev vs OpenAI's Decisions API
- [[ideas/head-to-head-small-models]] — Jev vs small and fast LLMs and local or open models
- [[ideas/head-to-head]] — Jev vs frontier LLMs, classic methods and cascades
- [[ideas/failure-reports-confidence]] — why a threshold does not carry between models
- [[ideas/cost-ledger]] — cost per decision

## Sources

Files in frontmatter `sources:`, captured by 2026-10-24; original URLs inline.