---
title: "Head-to-head benchmarks: reasoning, agent, robot and science suites"
type: community
source_tier: community
tags: [head-to-head, benchmarks, reasoning, agents, science, community]
created: 2026-10-19
updated: 2026-10-24
confidence: medium
sources: "54 files, listed under this page in https://jevwiki.ai/index.json (raw/ paths map to their URLs in https://jevwiki.ai/raw/MANIFEST.json); links are inline in the body"
jev_version: "jev-1.14.0"
summary: "Jev on reasoning, agent, robot, scientific and medical suites (Kepler, MedQA, WebArena, ALFWorld, GPQA, LLM Chess, System1-Arena, RealtimeBench): mostly mid-pack, cheap and fast."
---

# Head-to-head benchmarks: reasoning, agent, robot and science suites

> **TL;DR** Where the answer must be derived rather than read off the text, Jev is mid-pack, cheap and fast: 8th of 30 models on System1-Arena (behind Opus 5.5 at about 200× the cost, ahead of other hosted decision APIs), behind a 27B decision model and Gemma on 15 medical sets while near the top on format stability, third of 14 decision models on MedQA, and beaten inside an agent by a 4B model distilled from its own answers. A clinical-judging preprint (2026-10-20) put Jev 1.13 level with frontier LLM judges at 1/73-1/294 of their cost, behind the best open judges; on a live-clock web benchmark it was the fastest decider but reliably finished 8% of tasks to GPT-6-Astra's 36%. As the action picker in five agents it cut decision cost everywhere but lowered success on two mobile agents; in games and ALFWorld it improves when code supplies the prediction; on a live game clock it beat a reasoning LLM but trailed two rival decision models. Split from [[ideas/head-to-head-benchmarks]] on 2026-10-19 (rows unchanged); text classification, intent and legal sets stay there, with the reading notes.

Numbers as each builder published them, on Jev 1.13 (`jev-1.13.0`, or OpenRouter's `typesafe/jev-1.13-20260917`) unless stated; rows that name no version are assumed on it (inferred: `jev-latest` pointed at `jev-1.13.0` throughout 2026-09, [[reference/models-and-pricing]]). **LOST** = Jev lost on the stated metric. Every set here is public and old enough to be in some model's pretraining (several authors say so); one run each. Vendor or promotional sources are labelled. Own-set rows moved out on 2026-09-25: Sniff Test to [[ideas/head-to-head]], webctl to [[ideas/head-to-head-agents]].

## Structured and scientific data

| Dataset (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| NASA Kepler objects of interest, 8,054, 3 classes ([gist](https://gist.github.com/ipaulsmith/e5c3ae3a492a455435d5bfc161404312), ipaulsmith) | Jev Choice vs a fixed 3-rule baseline vs always FALSE POSITIVE | Raw measurements as `state` (20 raw fields plus a size bucket): **54.2%**, below rules **64.4%**; it picked CONFIRMED only 8 times. Code-bucketed named fields + structured criteria: 72.5% (CI 71.5-73.5), CANDIDATE recall 45.0%. 322 ms median, ~$0.34 | **LOST** raw; won bucketed | Round 2 designed after seeing round 1 and changed representation and instructions together, so it does not isolate bucketing (author); no held-out prompt set; no per-object results published. Matches [[concepts/jaggedness-jev-1-13]] on numbers: `verified` |
| Who&When Pro agent-failure attribution, 6,257 text traces ([TokenTrim](https://github.com/TokenTrim/jev-agent-failure-benchmark)) | Jev (3 Choices per trace) vs the paper's GPT-5.4, Claude Sonnet 4.6, GLM-5, Qwen3.5-122B | Who / when / what: Jev 73.4 / 76.4 / 23.7 vs GPT-5.4 55.7 / 72.3 / 15.3; next best "what" 22.2 (GLM-5); ~$1.28 total | "What" (the only like-for-like axis) | Jev picks from listed agents and steps while the LLMs free-generate; baselines are the paper's, not re-run; failures injected, not natural |
| ClinReg subset (Log10's clinical and regulatory benchmark): 1,000 held-out TrialPanorama evidence questions (4 options), TrialPanorama abstract screening (follow-up: 1,000 studies in 100 unused reviews, half eligible), 47 CLEF eHealth TAR full texts from one tuberculosis review ([Log10 Everest](https://everest.log10.io/news/evaluating-jev-for-answering-and-reviewing-clinical-research), Log10 Applied AI; [@log10io](https://x.com/log10io/status/2108214584968417362), 2026-10-08; **vendor**: Log10 sells Everest and runs ClinReg) | Three roles, each vs random selection: (1) Jev answers, Claude Opus replaces Jev's least confident 20%; (2) Jev alone judges saved answers from 26 models (Noul "Is this answer correct?"); (3) Jev picks the 20% of another model's answers Opus reviews, also vs the source model's own confidence | (1) Evidence: Jev 80.0%, cascade **87.9%**, random 20% 83.0% (+4.9, CI 3.6-6.2), Opus alone 94.9%; $5.46 vs $25.33 per 1,000 questions; Opus fixed 87 errors and added 8. Abstracts F1 70.7 → 74.9 (random 72.8). Full texts 39/47 → 43/47 (random 85.4%, Opus 95.7%), $0.64 vs $3.39. (2) On all three tasks below the rule "call every answer correct"; flags were real errors 36.7% (abstracts), 34.4% (full text), 11.5% (evidence; base error 11.0%). (3) Abstracts: 27.7 net corrections per source model vs random 10.0 and source confidence 18.7 (gap over source confidence: CI −1 to +19); full text 2.42 vs 0.80; evidence 13.04 vs 13.09 (no gain; Gemma 4 31B 70 vs 35, GPT-6 Astra −7 vs 2.2) | Cascade on answering; review picks on screening; **LOST** as sole judge | Selection used the chosen option's probability, not the `confidence` field; the abstract cascade and all full-text results are retrospective or exploratory (one 47-paper review); costs are estimated calls, no live latency. One confidently wrong full-text inclusion stayed outside the reviewed 20%. The authors suggest, untested, that Jev flags answers that disagree with its own pick. Their view: a limited role in these workflows; narrower decisions with explicit criteria may fit better |
| Clinical tasks in three answer forms: ten-year risk of death (NHANES 1999-2008) and death in intensive care (eICU demo) as probabilities; Choosing Wisely decisions on patient questions (plus MedQA and Medbullets board questions) as choices; colon cancer stage from pathology reports (and acute kidney injury stage) as ordered levels (medRxiv [10.64898/2026.10.09.26365130](https://www.medrxiv.org/content/10.64898/2026.10.09.26365130), Dean Ariel, Guy Laban, Stav Endelman, Ran Balicer, Noa Dagan, Seffi Cohen, posted 2026-10-12, preprint, not peer-reviewed; [code and aggregate results](https://github.com/deanariel-md/decision-model-offload); shared by [@OpenRoundsAI](https://x.com/OpenRoundsAI/status/2110411271678160963)) | Jev (`typesafe/jev-1.13`) vs five flagship LLMs (not named in the abstract) with standard LLM-style inputs; then Jev with the developer-recommended input changes (medical context, task decomposition); then a split: Jev keeps its fully confident staging answers, an LLM takes the rest | Standard inputs: mortality AUROC 0.822 vs 0.837-0.846; Choosing Wisely accuracy 87.7% vs 98.4-100%; staging 45.4% vs 94.2-100%. Staging with medical context 75.8%, with decomposition 82.1%, with both **85.8%**. None of Jev's staging errors had full confidence. The split was non-inferior to each LLM alone and cut flagship LLM cost by 33-44% | **LOST** alone on all three; split: cost | Abstract only (the body did not extract): LLM names, n per task, the exact inputs, the non-inferiority margin and the board-question results are not in our capture; patient questions and pathology reports are private (on request). A research comparison, not clinical guidance. The staging gain from added context and structure is in line with [[concepts/advanced-structure]] and [[concepts/how-to-build]] (`verified` in direction) |
| Clinical judging, four uses: 118 pediatric decision-support answers graded by five physicians on six rubrics and a 1-5 total; 29,511 HealthBench response-criterion pairs graded by physicians; MedHELM open-ended tasks (56 clinician-rated responses, 45,384 graded by its three-LLM jury); HealthBench Professional, 1,135 criteria against its GPT-5.4 grader (arXiv [2610.28826](https://arxiv.org/abs/2610.28826) v1, Rotem Gershon, Seffi Cohen and 12 coauthors, Clalit Research Institute, Ben Gurion University, Schneider Children's Medical Center and others, 2026-10-20; PDF only) | `jev-1.13.0` pinned: one Score per rubric carrying the original rubric text and anchors, read as the most probable level or as the expected score (levels weighted by their probabilities), and Nouls for pass or fail (yes at p ≥ 0.5); vs 12 LLM judges (Claude Opus 5.5, GPT-6, GPT-5, o3, MiMo-V2.6-Pro, GLM-5.3, gpt-oss-120b, Llama 4, GPT-4o and others) | Pediatric, Gwet's AC2 with physicians: Jev 0.50 (expected score 0.53) vs Opus 5.5 0.49, GPT-6 0.38; five judges higher (0.60-0.81, led by Llama 4, GPT-4o, GLM-5.3); on safety 0.78 vs the seven published judges' mean 0.71. HealthBench macro F1 0.665 vs gpt-oss-120b 0.667, GLM-5.3 **0.703**; on the half sample 0.664 vs Opus 5.5 0.578, GPT-6 0.535. MedHELM, 56 responses: ICC 0.49 (expected score) vs the jury 0.47. HealthBench Professional: same label as GPT-5.4 on 90.8% (other judges 90.9-93.3%). Cost per million judgments $38 / $54 / $89 (uses 1 / 2 / 4) vs Opus 5.5 $4,854-17,047; median 0.12-0.55 s per request vs 2.94-43.68 s. Hybrid (Jev judgments below 0.4 confidence go to an LLM): at most +0.04 agreement at 1.2-76× Jev's cost; routing only the safety rubric at a 0.2 cut raised caught unsafe answers from 31% to 49-51% | Cost and latency at frontier-judge agreement; **LOST** agreement to five judges (pediatric) and to GLM-5.3 and MiMo (HealthBench) | Preprint. Pediatric answers graded in English translation (on the Hebrew originals Jev's AC2 moved by at most 0.03); uses 3 and 4 mostly measure agreement with LLM graders; public sets may be in Jev's training (authors). Choice avoided because of the documented first-option lean (`verified`, [[concepts/jaggedness-jev-1-13]] mode 8). Jev's yes/no confidence ran lower than its ordinal confidence, so one cut deferred more (58% at 0.4). Several authors also wrote the offload preprint in the row above |
| Medicine: 15 datasets, 8,742 instances in four task families (clinical judgment, safety review, physiological assessment, triage and navigation); meaning-preserving format changes, a missing-right-answer test and optional abstention (arXiv [2610.18529](https://arxiv.org/abs/2610.18529) v2, Yanrui Du and eight coauthors, Harbin Institute of Technology and Ant Group, 2026-10-16) | Jev 1.13 and 1.14 vs LLMs (Gemma4-31B-IT, Qwen3.8-27B, Qwen3.6-27B, Kimi-K2.6) and Jev-like models (D3-27B, StartLux-27B, OpenJev, Kev-4B, GLiNER2.5-Decide and others, none of them Jev); 23 configurations | Overall accuracy: Jev 61.6-62.3% (macro-F1 58.7-59.7%) vs D3-27B **66.54%**, Gemma4-31B-IT 66.34%, StartLux-27B 64.83%. Clinical judgment 74.9-76.8% (Gemma 80.1, Qwen3.8 72.9); physiological assessment 35.9-39.4% vs Gemma 58.1; CTG-21 20.7-27.6%, below a 53.5% majority baseline; above 94% on ICD-proxy and MIMIC-CDM. Format changes moved Jev's accuracy by at most 0.45 points; decision reversals 5.69% (1.14) and 6.17% (1.13) vs Gemma 7.41%, Qwen3.8 10.75%, D3 4.74%. Swapping the right answer for none-of-these cost Jev 11.4-12.0 points; optional abstention (4.4-4.5% of items) raised retained accuracy only 0.3-0.5 points. Scoring each option as yes/no and normalising, instead of one Choice: at a 0.90 cut, 90.3-97.0% right vs 79.4-80.3%, but only 18.8-24.7% of items kept vs 39.9-42.3%; under Choice, 1.13 gave probability 1 to 1,819 decisions, 102 of them wrong | **LOST** overall to D3-27B and Gemma; near the top on format stability | Preprint; a research benchmark, not a basis for clinical use. Weak where numbers decide, as in [[concepts/jaggedness-jev-1-13]] (`verified` in direction). Per-option yes/no scoring helped Jev but lowered D3's and StartLux's accuracy at matched coverage: test it per model ([[ideas/failure-reports-confidence]]) |
| Which exam questions medical students get wrong: each model rates the difficulty of 300 multiple-choice medical questions, compared with students' real first-try results (Spearman) ([@0xnneo](https://x.com/0xnneo/status/2111247405773394360) thread: [cost](https://x.com/0xnneo/status/2111247519640367228), [fine-tune](https://x.com/0xnneo/status/2111247603924938763), [takeaways](https://x.com/0xnneo/status/2111247659113886122), 2026-10-17; chart transcribed; the authors' own data, not public) | Jev 1.14 and "Jev Router" vs Claude Opus, Sonnet and Haiku 5.5; Laya (not Jev) untuned and fine-tuned on 8,000 of the authors' questions; Gemini Argon on a 175-question subset | Opus 0.44, Jev 1.14 0.44, Jev Router 0.40, Sonnet 0.38, Haiku 0.33, Laya fine-tuned 0.23, untuned −0.04. On the subset: Gemini Argon 0.51, Opus 0.46, Jev 0.43. Jev 1.14 rated all 1,073 test questions for 2 cents in total, about 1/600 of Opus's price | Tie with Opus at far lower cost | Agreement is modest for every model: the best put only about half the questions in the right easy/medium/hard third, and real student data still beats them (authors). Laya fine-tuned is 0.23 on the chart, 0.30 in the thread. "Jev Router" is not explained (possibly OpenRouter's `typesafe/jev-router`, inferred). Jev 1.14 was `jev-preview` on these dates ([[reference/models-and-pricing]]) |

## Robots, browsers and reasoning suites

| Suite (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| Meta-World reach / push / pick-place, 2 seeds each; LIBERO drawer and microwave ([embodied-jev](https://github.com/FBddcz/embodied-jev), FBddcz) | Jev picks from code-built candidates (code does IK and safety) vs GPT-6 Astra | Meta-World, hierarchical (2026-09-21): Jev **5/6** = Astra 5/6 (both ran out of steps only on `push-v3-0`), estimated $0.018308 vs $3.605290. Direct four-channel control: Jev **2/6** vs GPT-6 5/6 (settings differ, not a clean ablation, our reading; per-arm budgets not in our capture; the earlier 40-step Jev pilots used a different budget and action interface). LIBERO (GPT-6 plans from dual-camera RGB-D, Jev picks the local moves): Jev acting was 81.7% cheaper ($0.39400 vs $2.15002) and 47.6% faster on the drawer; microwave: GPT-6 alone passed, the hybrid did not finish in 1,200 s (four runs in all: GPT-6 alone 2/2, hybrid 1/2) | Cost, tie on hierarchy; **LOST** direct control | Development runs; LIBERO shown records include reruns. Its failures: [[ideas/failure-reports]] |
| LIBERO microwave, drawer and alphabet soup, init state 0 ([jev-libero](https://github.com/Dimweaker/jev-libero), Dimweaker) | The simulator previews each of 27 inputs for up to 8 steps; only inputs meeting the task rules are offered; Jev picks intent, motion family, input | Microwave seeds 1 / 2 / 3: pass / fail / pass (14, 75, 53 decisions); seed 2 **stopped at 28.33° when no candidate met the rules**. Drawer pass (20 decisions, $0.001418). Alphabet soup failed first (kept lifting and lowering); after a config fix it passed containment, but release and settling were not shown. $0.012707 over 4 core runs | 3 of 4 core runs | Author: not a success rate. When code filters every option out, the run stops: plan a fallback |
| WebArena-Verified, a 12-task development subset of 812, official evaluator ([Jevry](https://github.com/michaelswissa/jevry), michaelswissa) | Jevry before and after fixes (an LLM plans, Jev picks actions) | 7/12 → **10/12** (an earlier run, `speed-001`, 9/12; tasks 47 and 102 still fail); median Jev request 443 → 392 ms, but median task time **23.367 → 44.169 s**: faster decisions did not remove repeated navigation | Accuracy up; slower | Repeatedly used subset, not a leaderboard result (author). A 449-option dropdown hit the 255-option Choice limit ([[reference/http-api]], [[concepts/choice]]: `verified`); a 375,564-character request was refused as `max_tokens_exceeded` (code not in TypeSafe's docs: `unverified`) |
| ALFWorld text (12 episodes) and Blackjack (100 hands), 2026-09-19 ([system1-agents](https://github.com/ThinkFlowLab/system1-agents), ThinkFlowLab) | Jev in the openJiuwen model slot vs a chat model (not recorded in the job folders) and a scripted reference (oracle plan; basic strategy) | ALFWorld 0.75 vs **0.917** (oracle plan 0.917), $0.0015 vs $0.0247 per episode (the author: 16 times the dollars). Blackjack: all three play the identical basic strategy (score −0.06 each), Jev at a fifth of the chat model's time per hand | Cost; Blackjack tie; **LOST** ALFWorld | Small n, 95% CIs overlap (0.50-1.00 vs 0.75-1.00); author's job folders. The README headline rests on single episodes: [[ideas/head-to-head-agents]]; Google Flights: [[ideas/builds-browser-and-interface]] |
| 180 GPQA + AIME rows; 856 rows pooled ([MetaCog](https://github.com/ItIsCuthNotCup/MetaCog), ItIsCuthNotCup) | An LLM writes reasoning paths, Jev Nouls judge each path and answer, more paths when unsure | 76.7% → **86.7%** (180 rows); 75.0% → 79.4% pooled. Jev Noul ≥ 0.95 right 226/229; right pick 224/245 when paths disagreed; Jev 72/127 vs a local logprob judge 68/127 (p = 0.50); Brier 0.16 | Judge added accuracy | The LLM does the maths, Jev only judges (`unverified`). McNemar tests and a per-row ledger published |
| RACE-H (reading comprehension) and MMBench (images turned into dense captions), multiple choice ([@Imroze_Aslam](https://x.com/Imroze_Aslam/status/2105057837764546859), 2026-09-29) | Jev vs LLMs and VLMs (not named in the text) | Within 1.3-2.4 accuracy points of models 10-40× its cost; "right on the Pareto frontier" on both | Cost; **LOST** accuracy narrowly | `unverified`: models, n and per-set numbers are in two charts, not transcribed; MMBench reached Jev only as captions |
| BIG-Bench Hard, 160 items over 4 tasks, 2026-09-20 ([jev-skill](https://github.com/wuyoscar/jev-skill), wuyoscar; [CALIBRATION_RESULTS.md](https://github.com/wuyoscar/jev-skill/blob/7154e950ee4b29ceebc236574c9dd6b7ac0dbe4a/evals/CALIBRATION_RESULTS.md)) | Jev vs DeepSeek V4.1 Flash (reasoning `none`), strict interface | 136/160 vs 120/160. `confidence` ≥ 0.90: 92/100 right, 8 wrong, 6 of them causal judgment (14/20 there). Top-probability bin [.9, 1]: 118 items, mean 0.9814, right 0.8898; middle bands not monotonic ([.70, .90) 70.4%, below .70 75.8%). Top probability ≥ .90 accepts 118 items, `confidence` ≥ .90 accepts 100. Model-panel rerun 2026-09-22 ([README](https://github.com/wuyoscar/jev-skill/blob/7154e950ee4b29ceebc236574c9dd6b7ac0dbe4a/docs/experiments/model-panel/README.md)): 138 vs 108 | Accuracy | The author calls the sample small and mixed: calibration `unverified` in general. Top probability and `confidence` are different signals (`verified`, [[concepts/confidence]]): don't swap them silently. Agent pilot: [[ideas/head-to-head-agents]] |
| LLM Chess leaderboard, 80 games; Komodo Dragon levels 1-3, 10 games each ([Maxim Saplin](https://dev.to/maximsaplin/typesafe-jev-played-chess-and-landed-next-to-reasoning-models-28ga)) | Jev picking from the code-built legal-move menu vs the board's LLMs and the engine | Elo 242.9 ± 117.5 (#59), beside o4-mini-medium 240.3 ($1.11 and 41.5 min per game); Jev $0.0015 and 35.6 s per game; every game completed. vs Dragon: 15 losses, 15 draws, 0 wins in 30 (5/5/0 per level). Our check 2026-09-24 of the unmerged `feat/typesafe-jev-player` branch's `_logs/`: 0 wins in 70 engine games (50 losses, 20 draws; levels 1-7; 20/20 lost at 6-7), 8 wins and 2 draws in 10 vs a random player | Cost, latency; Elo tie with mid-pack reasoning models; **LOST** to the engine | Wide Elo interval. No Jev row on the live leaderboard on 2026-09-24 (branch unmerged): `unverified` against a capture; the 70-game tally is our own count (our 2026-09-24 vetting record, not published), not in a raw capture. Zero illegal moves came from the legal-move menu, not judgment. [[ideas/failure-reports]] |
| Cognitive Reflection Test (150 new questions × 4 option orders), 210 random one-shot 3×3 matrix games, ALFWorld (140 seen, 134 unseen games), DeepMind Control Stacker (20 episodes, plus 5 new instructions × 8 seeds) (arXiv [2610.01834](https://arxiv.org/abs/2610.01834), "Code Owns the Simulation, Jev Owns the Evaluation", Yaodong Yang, Hongyao Tang, Yi Ma, Xingyu Fan, Weixun Wang, Jinpeng Li, Tianpei Yang; CUHK, Tianjin, Shanxi, Nanjing, CAIR-CAS; 2026-10-01; shared by [@SciFi](https://x.com/SciFi/status/2106904052168601760)) | `jev-1.13.0`, one Choice per call, zero-shot; conditions plain / told to reason / prediction written into `state` by code; code lookahead vs Jev alone; a hand-written controller, random choice, a word-overlap rule | CRT: right on **99%** of lure questions (intuitive answer 1%; published open-form human answers took the lure 55%), 96% on trap-free controls. Games: perfect with a dominant action; 77–93% when the payoff-average shortcut agrees with the equilibrium, **28–37%** when it misleads (lure 53–71%); a "reason first" instruction left that at chance; stating the opponent's move gave 97–99%. Asked separately Jev predicted the opponent right 90% of the time, yet chose right in 32% of calls with a right prediction vs 35% with a wrong one; predict, write into `state`, then decide: 33% → **94%**. ALFWorld steps (seen / unseen): task-completing steps 93 / 96%, enabling steps where the shortcut is right 99 / 95%, where it misleads **36 / 43%** (lure 56 / 51%). Next subgoal written in: 6% → 92% (goal-form tasks); code lookahead (each option shows the game's next output and newly possible commands) 36% → 79% at misleading steps. Whole unseen games: alone 42/134 (31%), with next output 79, plus new commands **116 (87%)**; a word-overlap rule with the same lookahead 18. Stacker: raw torques 0/20 (random 0/20); choosing among code-simulated skills 9/20 vs hand-written controller 8/20; 5 new instructions **22/40 vs 3/40** (random 0/40). ~94M input tokens, ~$4 | Evaluation when code supplies the simulation; **LOST** when one call must simulate and decide | Preprint, one model version. CRT human and GPT figures are published open-form answers, reference only (authors). The ThinkFlowLab ALFWorld row above is a separate 12-episode run. $4 for ~94M tokens fits $0.042/M ($3.95, our arithmetic: `verified`, [[reference/models-and-pricing]]). Repeat noise it reports: [[ideas/request-mechanics]]; lesson: [[ideas/failure-reports-wrong-role]] |
| Online-learning game: 4 actions × 64 rounds, 9 loss sequences fixed before play, every action's loss history in `state` ([@ArdhenduTripat9](https://x.com/ArdhenduTripat9/status/2107144705498575117) thread, 2026-10-05; tables transcribed; code in a Colab notebook, not captured) | Jev Choice probabilities used as allocation weights vs follow-the-leader (FTL), Hedge, OMD, GPT-6 Luna (medium; one game per row); then a Noul per action ("will it take a loss next round?") turned into weights in code by argmin or softmin | Static regret, lower is better. Choice: lower than FTL on 7 of 9 games and best of all on two stable-winner games (2.36, 0.41), but 22.52 on noisy alternation and 21.94 when the past leader is penalised (best: OMD −1.49, Hedge 0.84). Noul forecasts: perfect alternation 2.60 → **−27.00** argmin / −20.73 softmin (Luna −13.60); noisy alternation 22.52 → 3.50 / 1.08; past leader penalised 21.94 → 4.00 / 2.41; equal means 10.32 → 7.50 / 4.86 (Hedge 3.36). Softmin below Luna on 7 of 9 (our count) | Forecast-first beat Luna on most games; **LOST** as a direct Choice wherever the leader changes | Author: Choice weighting behaves like follow-the-leader; separating per-action prediction from allocation in code fixed most of it. Synthetic sequences, one game per row; `unverified`. Numeric sequences are a documented weak spot ([[concepts/jaggedness-jev-1-13]]); weighing in code matches that advice |
| MMLU-Pro (12,032), ARC-Challenge (1,172), GPQA Diamond (196), HLE multiple-choice subset (494), MATH-500 as options (261) and as per-digit read-out (273), ARC-AGI-2 public set (167 grids / 120 tasks): 16,379 planned requests in three frozen suites, September 2026 ([JevResearch/Jev-Research](https://github.com/JevResearch/Jev-Research), [report](https://jevresearch.github.io/Jev-Research/report/); README says revision 3, the page revision 5, both 2026-10-03; unaffiliated; secondary write-up by [remio](https://www.remio.ai/post/jev-benchmark-finds-a-useful-decision-model-not-a-frontier-reasoner)) | Service-reported `jev-1.13.0`, one shot, no retries, failures scored wrong ("all-requested"); vs 15 OpenRouter models run by the authors on the same frozen items (README says 12; MMLU-Pro on a 1,000-item stratified subset), vals.ai published scores (positioning only), Qwen3.5 9B (XNLI, political questions) | MMLU-Pro **82.7%** (CI 82.0–83.4; probability-weighted 74.0%), ARC-Challenge 97.9%, GPQA Diamond **76.5%** (70.1–81.9; weighted 63.0%), HLE MC **21.9%**, MATH-500 83.1% as options vs **13.6%** digit by digit (0.817 vs 0.103 on the 224 shared items), ARC-AGI-2 **0/120** exact tasks (per-cell 59.6–61.0%, a diagnostic encoding). Calibration ECE: MMLU-Pro 0.050, GPQA 0.088, HLE **0.415** (mean selected probability 0.633 at 21.9% right); gold answer at zero probability on 213 MMLU-Pro items. Shuffled options flipped 5.3% of MMLU-Pro answers, no net change. 19 contract-invalid responses in ~16k. Matched subset, MMLU-Pro (Jev 84.0% on those 1,000): GLM-5.3 88.0, DeepSeek V4 Flash 87.4, Qwen3.8 Max 87.2, Qwen3.8 Flash 86.3, GLM-5.3 Flash 85.4, MiMo v2.6 Pro 85.2, Qwen3.7 Flash 84.2, MiMo v2.6 Flash 84.0, gpt-oss-120b 81.9, gpt-oss-20b 74.7, Mistral Small 3.2 57.7, Mistral Nemo 36.9, Granite 4.0 H Micro 32.9, Llama 3.1 8B 31.9, Gemma 3 4B 31.4; GPQA: 7 of 15 above Jev (best GLM-5.3 87.8); HLE: 4 of 12 above (best MiMo v2.6 Flash 34.8). Provider-reported cost of those 1,000 MMLU-Pro items $0.0064 (Llama 3.1 8B) to $10.76 (GLM-5.3); Jev's whole 12,032-item run **$0.28**. XNLI, 1,200 premises in 6 languages: Jev beat Qwen3.5 9B (no reasoning) in all six (English 87.5 vs 83.3%; others 77.4–82.4 vs 71.3–76.4%). Politically sensitive closed-book questions: Jev did not repeat Qwen's PRC-line denials (Taiwan's administering government: Taipei 20/20 vs Beijing 20/20) | Knowledge multiple choice at a tiny cost; **LOST** multi-step (HLE, ARC-AGI-2, digit read-out) and to 7 matched cheap models on the MMLU-Pro subset and GPQA | `unverified`: aggregates published, raw responses and licensed items not, so re-rendering the page is not re-scoring (authors). Published comparison scores mostly use reasoning and few-shot prompts. GPQA differs from the 78.3 Cloudflare reported for Jev and MMLU-Pro coincides ([[ideas/rival-decision-models]], separate runs); Archer Hume's MMLU-Pro 84.6% is another sample ([[ideas/request-mechanics]]). Baselines ran at provider defaults, except five cells kept from an earlier temperature-0, low-effort protocol (all four Granite cells, Qwen3.8 Max on MMLU-Pro), with 1,157 answer-parsing recoveries. $0.28 / 12,032 ≈ 550 input tokens per question at $0.042/M (our arithmetic), plausible. Knowledge read strongest on 2024-era facts; size and architecture are their inference. Latency, rounding, option order, repeat noise and `confidence`: [[ideas/request-mechanics]]; small-model reading: [[ideas/head-to-head-small-models]] |
| ThinkFast no-CoT suite (14 benchmarks), nocot-bench's NCRI ladder (279 models), three transcript-monitoring sets: coding sabotage, SHADE-Arena action-only, sandbagging ([LessWrong](https://www.lesswrong.com/posts/QW2kPrAyZQvRtBXHG/benchmarking-jev-against-no-cot-llms), Dewi Gould and James Mann, 2026-10-07; summarised by [@jevwatch](https://x.com/jevwatch/status/2107934790154633325)) | Jev 1.13 vs GPT-5.5, Opus 4.7 and Opus 4.1 answering with no chain of thought. Monitors: the benchmark's instructions plus one Score whose levels come from its rubric, suspicion = probability-weighted level. Multi-part answers built from one question per part (a sudoku = 81 Choices). NCRI's free-form questions turned into option lists (up to 255 consecutive integers) | ThinkFast mean, chance-corrected: Jev 46.8% vs GPT-5.5 58.1%, Opus 4.7 53.7%, Opus 4.1 40.4%. GPQA-Diamond **65.2** vs GPT-5.5 51.8; within 5 points on N-hop lookup, CyBash, training-poisoning; behind on BEA-24, intuitive physics, Sally-Anne; weakest on multi-part answers (sudoku, test-case pass/fail, chess, Tower of London). Monitor AUROC: coding sabotage 0.99 (GPT-5.5 and Opus 4.7 0.99, Opus 4.1 0.74), SHADE-Arena 0.90 (0.93, 0.92, 0.86), sandbagging **0.94** (0.91, 0.97, 0.86); caught at a 1.5% false-positive rate: 89% vs 93–94%, 47% vs 67–71%, 55% vs 68–76%. NCRI 49.7 [45.2, 53.9], rank 209 of 279, beside GPT-3.5-turbo-instruct and GPT-4.1-nano (rank 38, 108, 154 with 5, 20, 50 integer options). Cost about 1/200 of Opus 4.7's | GPQA-Diamond; monitor AUROC near the frontier at a fraction of the cost; **LOST** catches at a strict false-alarm budget, multi-part answers and multi-step reasoning | Authors: Jev is "very jagged"; cheap enough for routing and always-on agent monitoring, but it leaves no reasoning trace to audit. NCRI rank is an upper bound (picking from a list is easier, and the LLMs answered free-form). Jev read only transcripts within its limit (134/511 SHADE, 297/379 coding, 244/408 sandbagging; all models scored on that subset): 32k per question `verified` ([[reference/models-and-pricing]]); 255 options `verified` ([[reference/http-api]]). The post's summary gives other 1.5%-rate figures than its monitor section (88.0 vs 90.8, 45.9 vs 59.3, 41.3 vs 63.5); @jevwatch's "46% vs 59%" is the summary's. Its parameter guess (fit at about 17B) is, by its own account, rough at best |
| 13 multiple-choice benchmarks: knowledge (MMLU-Pro, MMLU-Redux, MMLU 14,042, GPQA Diamond 198, C-Eval, HellaSwag, WinoGrande, TruthfulQA 817), reasoning (ARC-Challenge 1,172, MathQA, AQuA-RAT 254), multilingual (MMLU-ProX, MMMLU) (arXiv [2610.11978](https://arxiv.org/abs/2610.11978), Xing Li, Qingcheng Chang, Jinzhong Ning, Changfeng Xu, Shenlong Zhang, Yijia Zhang, Ling Luo, Hongfei Lin; Dalian University of Technology; v1 2026-10-08) | Jev (one Choice per item) vs 19 LLMs: six frontier (GPT-5.6 Sol, Claude Opus 5.5, Gemini 3.8 Flash, Qwen3.8-Flash and -Max, DeepSeek-V4.1-Flash), seven representative, six small (< 10B) | Best of all 20 on MMLU-Redux 95.0 and ARC-Challenge 97.6; vs the frontier median TruthfulQA +4.5, WinoGrande −0.3, MMLU −1.2, MMLU-Pro −5.8, C-Eval −8.3, GPQA −8.9, AQuA-RAT −7.9, MathQA **75.0 vs 92.7 (−17.7)**, below all 19. Outside maths ahead of the representative tier in 68 of 76 comparisons and of the small tier in all 62 | Knowledge and commonsense; **LOST** maths word problems to every model, Chinese-exam knowledge (C-Eval) | Authors' numbers, preprint. Jev and frontier models one run each; 104 of 169 smaller-model scores are published figures with other prompts (authors: rough comparison only). The maths gap fits documented failure mode 2, keep arithmetic in code (`verified`, [[concepts/jaggedness-jev-1-13]]). Close to arXiv 2609.37647 (Bonn suite above) on MMLU and WinoGrande (≤ 0.5 points) and C-Eval (1.4); HellaSwag 6.2 lower, likely the option template (authors; failure mode 8) |
| MedQA (USMLE, 4 options), all 1,273 test questions, 1 attempt each, 14 decision models through OpenRouter, every request within 11 minutes from one residential IP in Tokyo ([@zarate_ai](https://x.com/zarate_ai/status/2109310911223869774), 2026-10-11; 4 charts transcribed) | Jev 1.13 vs 13 rival decision models (none is Jev: [[ideas/rival-decision-models]]): Perplexity Decider v1.1 27B, Inception Mercury Decide, LiquidAI d1, OpenAI GPT-6 Luna Decisions, Microsoft Decision-1, Cloudflare Clef, Clef Omni and Clef Flash, Nace.AI Drex v1.5, Together Tev1 4B Experimental, Upstage Solar Decide and Decide Flash, Kev 4B | Accuracy: Decider 88.1%, **Jev 87.7%** = Luna Decisions, d1 87.0%, Mercury 85.2%, Clef 84.9%, the rest 67.1-83.7%. Cost of all 1,273 (median billed cost per question × 1,273): Mercury $0.0076, Decider $0.0078, Jev $0.0293, Luna $0.0412, Clef $0.1124. Median latency: Clef Omni 0.70 s, **Jev 0.71 s**, Luna 0.74 s, the rest 0.81-1.47 s. Calibration: Jev ECE 0.01 (lowest, with Clef), AUROC 0.90 (d1 0.91). The author's rank points: Mercury 35, Decider 34, **Jev 32** (3rd of 14) | Latency, calibration; **LOST** cost to Mercury and Decider, accuracy by 0.4 pts to Decider | One run; the top models' accuracy intervals overlap (author); public set, so pretraining overlap is possible; the points scheme is the author's; Luna Decisions refused 3 (counted wrong); latency is the client's round trip through OpenRouter. $0.0293 implies ~550 input tokens per question at $0.042/M (our arithmetic; plausible, token counts not published) |
| ALFWorld text (134 valid-unseen tasks) and AndroidWorld (116 task types, text UI), 5 agents (ReAct, Reflexion; T3A, AutoDroid, FastAgent), 3 runs each, 4,500 executions (arXiv [2610.13092](https://arxiv.org/abs/2610.13092), "When Faster Decisions Do Not Make Better Agents", Yuchen Ling, Shengcheng Yu, Zhenyu Chen, Chunrong Fang; Nanjing University, TU Munich; v1 2026-10-09; via [@ValesyaAI](https://x.com/ValesyaAI/status/2109540515536425418)) | Each agent's own `gpt-5.6-luna` action picker vs `jev-1.13.0` picking from candidates, one Jev request per step, Luna kept for open text and reflection; on ALFWorld also Luna given Jev's candidate list | Success: ALFWorld ReAct 96.02% (−1.49 pts; −1.00 vs Luna on the same list), Reflexion 98.01% (−1.74; 0.00); AndroidWorld T3A −13.22, AutoDroid −2.30, FastAgent **−18.10** pts (T3A and FastAgent intervals wholly below zero). Per decision: cost −66.1 to −78.4%, time −73.9 to −81.5% (4.13-8.13 s → 1.04-1.75 s), tokens +53.5 to +134.9%. Per task: cost −17.1 to −68.8%; time −47.6% (ReAct) to +0.7% (FastAgent). Cost per success fell for 4 agents, rose 7.7% for T3A (68.70 → 73.97 mUSD; interval spans zero); FastAgent time per success 239.4 → 377.8 s. Reflexion: first trial 383 vs 376 of 402 right, but recovered 11 of 19 failures vs 25 of 26 | Decision cost and speed; **LOST** success on two mobile agents | Preprint; one seed on fixed tasks; text-only inputs to suit Jev. Traces: failed recovery and "done" with the goal unmet ended runs. Not the same work as ThinkFlowLab's 12-episode row or arXiv 2610.01834 above. Authors: judge by cost per completed task, not per decision; agent A/Bs: [[ideas/head-to-head-agents]] |
| Reinforcement learning: nine MiniGrid tasks and three Atari games (Seaquest, Freeway, SpaceInvaders), DQN students, three training seeds (arXiv [2610.11692](https://arxiv.org/abs/2610.11692), Yi Ma, Tianpei Yang, Yaodong Yang, Weixun Wang, Hongyao Tang; [@Trtd6Trtd](https://x.com/Trtd6Trtd/status/2110143550142988702), 2026-10-13) | Frozen Jev as a zero-shot policy vs Qwen3-8B (non-thinking, same structured input); then as reference policy, exploration judge and replay rater vs plain DQN and prioritized replay | Direct control: Jev matched or beat Qwen3-8B on all nine MiniGrid tasks; Atari mixed. Jev's success with vs without engineered navigation hints: FourRooms 96 → 40%, MultiRoom-N4 100 → 38%, DoorKey-8×8 100 → 100%. Training with Jev beat plain DQN on several tasks, including some where DQN alone made no progress; no configuration was best everywhere, and Jev-rated replay did not consistently beat prioritized replay (most results are figures, not captured as numbers) | As a prior and guide; **LOST** as a long-horizon controller on its own | Jev's inputs carried information the student lacked (structured descriptions, task rules, route features). The authors rule out the value-function role because its update does arithmetic on the answer, which fits failure mode 2 ([[concepts/jaggedness-jev-1-13]]). Three seeds |
| System1-Arena: 92k questions across 105 tasks from 268 public sources (classification and knowledge, applied decisions, evaluation and control, agent and action) plus 138 closed-loop environments (web shopping, SWE-bench coding control, robotics, games, driving; some in real time) (arXiv [2610.21244](https://arxiv.org/abs/2610.21244), Tianyu Chen, Xinyang Liu, Ruyi An, Zhendong Wang, Mingyuan Zhou, UT Austin, v1 2026-10-15; [code](https://github.com/TianyuCodings/S1-Arena), [leaderboard](https://tianyucodings.github.io/S1-Arena); relayed by [@neuralloot](https://x.com/neuralloot/status/2111552937474597359)) | 30 models on the same decision requests: 7 closed LLMs (verbalized probabilities), 13 open LLMs read at the answer token, 10 System-1 models incl. Jev 1.13, Microsoft-Decision-1, the OpenAI Decisions API and open Decision-2.0 models (not Jev) | Table 1 (78,362 questions over 87 tasks): Claude Opus 5.5 most accurate, 81.1% at $10.38 per 1,000 requests and 2,293 ms; Jev 1.13 72.6% (8th of 30), Brier 0.349, 321 ms, $0.052 per 1,000, so Opus costs about 200× more (our arithmetic); Gemma-4-31B read at the answer token 73.9%, Decision-2.0-Vega-27B 73.0%, GPT-6 Luna 71.2% at $0.14, Microsoft-Decision-1 70.9% at $0.038, OpenAI Decisions API 68.6%. Agent steps (3,481) and closed loops (127 environments): Jev step accuracy 67.8% with no answer on 13.2% of steps, loop success 17.4% vs Vega-27B 26.5%; best loop success 32.9% (a large open LLM) | **LOST** to Opus and other frontier LLMs; first among hosted System-1 APIs | Preprint. Labels inherited from public, partly old sources (possible training overlap); hosted latencies include the network; one draw per request. The abstract's 92k / 105 / 138 describe the whole benchmark, the tables 78,362 / 87 and 127 (as @neuralloot quotes) |
| Cognitive and social biases: 41,500 responses over cognitive-reflection tasks, five classic bias paradigms and a pricing test for ethnic and gender bias, then 15,000 more over 30 conjunction tasks (arXiv [2610.19619](https://arxiv.org/abs/2610.19619), Ilaria Imperatrice, Taisia Kameneva, Adam Eric Greenberg, Alessandro Romano, Bocconi University, v1 2026-10-14) | Jev 1.13 against the human patterns each paradigm is known for | Chosen answers mostly avoid the human bias patterns; cognitive-reflection problems answered very well; small demographic gaps in the pricing scenario. But answers moved with irrelevant numbers and with option order, and bias manipulations shifted returned probabilities even when the pick held. A well-known probability problem: right in every response; an adapted version: the same rule broken in every response. Across 30 conjunction tasks the rule was broken in 54% of responses, accuracy swinging with wording and answer order | Picks mostly unbiased; **LOST** on variants and probabilities | Preprint; numbers from the abstract. Option order and number sensitivity match [[concepts/jaggedness-jev-1-13]] (`verified` in direction); test your own variants, not only famous items |
| RealtimeBench: 180 tasks over six game types (Mario, Snake, Tetris, Temple, River, Bird); the game keeps running while the agent decides; text observations and harness-built legal actions (arXiv [2610.20665](https://arxiv.org/abs/2610.20665), "The World Won't Wait", Ningxin Pan and five coauthors, Peking University and Samsung Research Beijing, v1 2026-10-14) | Jev (version not stated) vs GPT-6.1-sol (medium reasoning, ~19 s per response), other LLMs and decision models (Clef 27B and Clef-flash, StartLux 4B and 27B, GLiClass, replicas; none of them Jev) | Paused game: GPT-6.1-sol ahead in five of six game types. Live clock, mean progress 0-100: Jev 13.51 vs GPT-6.1-sol 1.51, but StartLux 27B 16.62 and Clef 27B 16.34 beat Jev; Jev 0.52 s per response, $0.0111 per task. TORSI (a slow teacher writes reusable instructions between attempts, Jev acts with no teacher calls) raised Jev from 13.51 to 24.19 in two rounds, in all six game types | Beat the reasoning LLM live; **LOST** to two rival decision models | Preprint; ten test tasks per game type, one run each mostly. Other game comparisons: [[ideas/head-to-head-games]] |
| Realtime-Bench (not RealtimeBench above): 50 tasks, 30 web (live auctions, room holds, a market-dip purchase) and 20 games, where the world advances by each model's real latency (Wall Clock); 3 runs (arXiv [2610.22155](https://arxiv.org/abs/2610.22155) v3, Jingyu Tang and nine coauthors, UT Austin, Ohio State, UIUC, NVIDIA, Northwestern, Steadyworks, v1 2026-10-15; [@JingyuTang621](https://x.com/JingyuTang621/status/2113678765960405272) thread, 2026-10-23; chart transcribed) | Jev 1.13 and OpenAI Decisions via OpenRouter, one Choice over harness-built GUI actions (Jev reads an accessibility-tree text view; OpenAI Decisions also gets screenshots), vs GPT-6-Astra, GPT-5.6 Sol, Claude Fable 5.1, Claude Opus 5, Gemini 3.8 Flash, DeepSeek V4.1 Flash, Kimi K3 and three real-time streaming models (GPT Realtime 2, Gemini 3.1 Flash Live, Gemini 3.8 Live) | Pass^3 (all three runs succeed): GPT-6-Astra (high) **36%** at 3.99 s per decision; Gemini 3.8 Flash 14-22%; Fable 5.1 and Opus 5 (low) 10%; Jev **8%** at 0.47 s, the fastest; GPT Realtime 2 6%; OpenAI Decisions 4% at 1.13 s; the rest 2-6%. Jev's runs 16 / 10 / 10% vs Astra 42 / 42 / 44% | Latency; **LOST** success to Astra, Gemini 3.8 Flash and the Claude models; ahead of OpenAI Decisions | Jev ran only the 30 web tasks (its text view misses game state), yet its rates are multiples of 2%, so they appear to be over all 50 with the 20 games as misses (inferred): read 8% as a floor for web-only use. Both decision models ran on the wall clock only. A failure example: Jev bought after the price had rebounded past the limit, on a 0.4 s call. Preprint |
| AgentDecisionBench: decision models from isolated questions up to control of a running agent, on public benchmarks and the authors' DecisionGym; 500 tasks for the harness test (arXiv [2610.21412](https://arxiv.org/abs/2610.21412), "JevAgentHarness", Kunlun Zhu and seven coauthors, UIUC and others, v1 2026-10-15; relayed by [@dair_ai](https://x.com/dair_ai/status/2111496216567742892); models and harness promised at github.com/JevAgentHarness) | Jev 1.13 (pinned `typesafe/jev-1.13`) and other decision models taking the LLM's routine choices (tool, arguments) vs the LLM deciding everything; AgentDecision-4B, an open 4B model trained on states the agent reaches after its own mistakes with Jev's answers as labels (not Jev) | The decision model most accurate on isolated questions solved only 69.5% of agent tasks; delegating tool choices without a fitted confidence threshold cost 19.0 points of success (a fixed 0.5 threshold: −21.5). Thresholds fitted per choice on the agent's own trajectories let tool and argument cascades cut cost 14.2% without changing the outcome of any of 500 tasks. On tasks unlike the training data the agent solved 91.5% with AgentDecision-4B vs 82.0% with Jev; on familiar tasks such 4B models made agents 81.1-89.0% cheaper and up to 2.5× faster, within 3 points of success | **LOST** to a distilled 4B out of distribution; Jev-gated harness level on outcome at lower cost | Preprint. Below the threshold the LLM decides; a fixed threshold is the main failure. Gates: [[ideas/patterns-agents]]; other agent A/B tests: [[ideas/head-to-head-agents]] |
| Multi-agent workflows: routing and review take 31.7-43.5% of generative-LLM tokens in three profiled frameworks; four benchmarks, two LLM backbones (arXiv [2610.21411](https://arxiv.org/abs/2610.21411) v2, "Jev-LiteMAS", Kaixiang Wang, Zhaijiacheng Zhou, Jiong Lou, Jie Li, 2026-10-16) | Jev picks the next expert from the task and earlier experts' outputs and screens intermediate artifacts (passing ones go on, flagged ones get an LLM re-review) vs the LLM-only baseline (LAS) | Generative-LLM tokens down 12.7-54.6%, accuracy within 1.34 points, lower latency in seven of eight settings; grounding routing in the experts' actual outputs avoided up to 18.0 points of accuracy loss | Cost at near-equal accuracy | Preprint; abstract numbers; Jev version not stated. LLMs still plan and write repair feedback; Jev only decides |

NARCBench multi-agent collusion, Jev black-box monitor vs vote counting and white-box probes (jev-collusion-monitor): [[ideas/builds-agents]].

## How to read these

The reading notes on [[ideas/head-to-head-benchmarks]] apply here too (public is not unseen, option order moves answers, thresholds are per dataset, check simple baselines). Numbers belong in code ([[concepts/jaggedness-jev-1-13]]). Patterns: [[ideas/patterns-agents]] P04 loop control, [[ideas/patterns-data]] P24 real-time; [[ideas/patterns-emerging]] P41 engine owns the truth.

## Related

- [[ideas/head-to-head-benchmarks]]: text classification, intent and legal sets; [[ideas/head-to-head-agents]]: agent A/Bs; [[ideas/head-to-head-games]]: games; [[ideas/eval-boards]]: boards; [[ideas/cost-ledger]]
- [[concepts/confidence]], [[guides/testing-and-evaluation]]: run your own comparison

## Sources

Links inline; raw captures listed in the frontmatter. The llm_chess 70-game tally comes from our own 2026-09-24 vetting of the harness branch (our 2026-09-24 vetting record, not published), not a capture.