Agents: read the raw Markdown of this page, or start at llms.txt.
~/wiki/ideas
Head-to-head benchmarks: reasoning, agent, robot and science suites
TL;DR Where the answer must be derived rather than read off the text, Jev is mid-pack, cheap and fast: 8th of 30 models on System1-Arena (behind Opus 5.5 at about 200× the cost, ahead of other hosted decision APIs), behind a 27B decision model and Gemma on 15 medical sets while near the top on format stability, third of 14 decision models on MedQA, and beaten inside an agent by a 4B model distilled from its own answers. A clinical-judging preprint (2026-10-20) put Jev 1.13 level with frontier LLM judges at 1/73-1/294 of their cost, behind the best open judges; on a live-clock web benchmark it was the fastest decider but reliably finished 8% of tasks to GPT-6-Astra's 36%. As the action picker in five agents it cut decision cost everywhere but lowered success on two mobile agents; in games and ALFWorld it improves when code supplies the prediction; on a live game clock it beat a reasoning LLM but trailed two rival decision models. Split from Head-to-head benchmarks: Jev on public datasets and suites on 2026-10-19 (rows unchanged); text classification, intent and legal sets stay there, with the reading notes.
Numbers as each builder published them, on Jev 1.13 (jev-1.13.0, or OpenRouter's typesafe/jev-1.13-20260917) unless stated; rows that name no version are assumed on it (inferred: jev-latest pointed at jev-1.13.0 throughout 2026-09, Models, aliases, pricing, rate limits, context). LOST = Jev lost on the stated metric. Every set here is public and old enough to be in some model's pretraining (several authors say so); one run each. Vendor or promotional sources are labelled. Own-set rows moved out on 2026-09-25: Sniff Test to Head-to-head: Jev against other models and methods, webctl to Head-to-head: Jev inside agents, routers and tool gates.
Structured and scientific data
| Dataset (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| NASA Kepler objects of interest, 8,054, 3 classes (gist, ipaulsmith) | Jev Choice vs a fixed 3-rule baseline vs always FALSE POSITIVE | Raw measurements as state (20 raw fields plus a size bucket): 54.2%, below rules 64.4%; it picked CONFIRMED only 8 times. Code-bucketed named fields + structured criteria: 72.5% (CI 71.5-73.5), CANDIDATE recall 45.0%. 322 ms median, ~$0.34 |
LOST raw; won bucketed | Round 2 designed after seeing round 1 and changed representation and instructions together, so it does not isolate bucketing (author); no held-out prompt set; no per-object results published. Matches Jev 1.13 jaggedness: known failure modes on numbers: verified |
| Who&When Pro agent-failure attribution, 6,257 text traces (TokenTrim) | Jev (3 Choices per trace) vs the paper's GPT-5.4, Claude Sonnet 4.6, GLM-5, Qwen3.5-122B | Who / when / what: Jev 73.4 / 76.4 / 23.7 vs GPT-5.4 55.7 / 72.3 / 15.3; next best "what" 22.2 (GLM-5); ~$1.28 total | "What" (the only like-for-like axis) | Jev picks from listed agents and steps while the LLMs free-generate; baselines are the paper's, not re-run; failures injected, not natural |
| ClinReg subset (Log10's clinical and regulatory benchmark): 1,000 held-out TrialPanorama evidence questions (4 options), TrialPanorama abstract screening (follow-up: 1,000 studies in 100 unused reviews, half eligible), 47 CLEF eHealth TAR full texts from one tuberculosis review (Log10 Everest, Log10 Applied AI; @log10io, 2026-10-08; vendor: Log10 sells Everest and runs ClinReg) | Three roles, each vs random selection: (1) Jev answers, Claude Opus replaces Jev's least confident 20%; (2) Jev alone judges saved answers from 26 models (Noul "Is this answer correct?"); (3) Jev picks the 20% of another model's answers Opus reviews, also vs the source model's own confidence | (1) Evidence: Jev 80.0%, cascade 87.9%, random 20% 83.0% (+4.9, CI 3.6-6.2), Opus alone 94.9%; $5.46 vs $25.33 per 1,000 questions; Opus fixed 87 errors and added 8. Abstracts F1 70.7 → 74.9 (random 72.8). Full texts 39/47 → 43/47 (random 85.4%, Opus 95.7%), $0.64 vs $3.39. (2) On all three tasks below the rule "call every answer correct"; flags were real errors 36.7% (abstracts), 34.4% (full text), 11.5% (evidence; base error 11.0%). (3) Abstracts: 27.7 net corrections per source model vs random 10.0 and source confidence 18.7 (gap over source confidence: CI −1 to +19); full text 2.42 vs 0.80; evidence 13.04 vs 13.09 (no gain; Gemma 4 31B 70 vs 35, GPT-6 Astra −7 vs 2.2) | Cascade on answering; review picks on screening; LOST as sole judge | Selection used the chosen option's probability, not the confidence field; the abstract cascade and all full-text results are retrospective or exploratory (one 47-paper review); costs are estimated calls, no live latency. One confidently wrong full-text inclusion stayed outside the reviewed 20%. The authors suggest, untested, that Jev flags answers that disagree with its own pick. Their view: a limited role in these workflows; narrower decisions with explicit criteria may fit better |
| Clinical tasks in three answer forms: ten-year risk of death (NHANES 1999-2008) and death in intensive care (eICU demo) as probabilities; Choosing Wisely decisions on patient questions (plus MedQA and Medbullets board questions) as choices; colon cancer stage from pathology reports (and acute kidney injury stage) as ordered levels (medRxiv 10.64898/2026.10.09.26365130, Dean Ariel, Guy Laban, Stav Endelman, Ran Balicer, Noa Dagan, Seffi Cohen, posted 2026-10-12, preprint, not peer-reviewed; code and aggregate results; shared by @OpenRoundsAI) | Jev (typesafe/jev-1.13) vs five flagship LLMs (not named in the abstract) with standard LLM-style inputs; then Jev with the developer-recommended input changes (medical context, task decomposition); then a split: Jev keeps its fully confident staging answers, an LLM takes the rest |
Standard inputs: mortality AUROC 0.822 vs 0.837-0.846; Choosing Wisely accuracy 87.7% vs 98.4-100%; staging 45.4% vs 94.2-100%. Staging with medical context 75.8%, with decomposition 82.1%, with both 85.8%. None of Jev's staging errors had full confidence. The split was non-inferior to each LLM alone and cut flagship LLM cost by 33-44% | LOST alone on all three; split: cost | Abstract only (the body did not extract): LLM names, n per task, the exact inputs, the non-inferiority margin and the board-question results are not in our capture; patient questions and pathology reports are private (on request). A research comparison, not clinical guidance. The staging gain from added context and structure is in line with Structured instructions, options, levels, criteria and How to build software with System One (verified in direction) |
| Clinical judging, four uses: 118 pediatric decision-support answers graded by five physicians on six rubrics and a 1-5 total; 29,511 HealthBench response-criterion pairs graded by physicians; MedHELM open-ended tasks (56 clinician-rated responses, 45,384 graded by its three-LLM jury); HealthBench Professional, 1,135 criteria against its GPT-5.4 grader (arXiv 2610.28826 v1, Rotem Gershon, Seffi Cohen and 12 coauthors, Clalit Research Institute, Ben Gurion University, Schneider Children's Medical Center and others, 2026-10-20; PDF only) | jev-1.13.0 pinned: one Score per rubric carrying the original rubric text and anchors, read as the most probable level or as the expected score (levels weighted by their probabilities), and Nouls for pass or fail (yes at p ≥ 0.5); vs 12 LLM judges (Claude Opus 5.5, GPT-6, GPT-5, o3, MiMo-V2.6-Pro, GLM-5.3, gpt-oss-120b, Llama 4, GPT-4o and others) |
Pediatric, Gwet's AC2 with physicians: Jev 0.50 (expected score 0.53) vs Opus 5.5 0.49, GPT-6 0.38; five judges higher (0.60-0.81, led by Llama 4, GPT-4o, GLM-5.3); on safety 0.78 vs the seven published judges' mean 0.71. HealthBench macro F1 0.665 vs gpt-oss-120b 0.667, GLM-5.3 0.703; on the half sample 0.664 vs Opus 5.5 0.578, GPT-6 0.535. MedHELM, 56 responses: ICC 0.49 (expected score) vs the jury 0.47. HealthBench Professional: same label as GPT-5.4 on 90.8% (other judges 90.9-93.3%). Cost per million judgments $38 / $54 / $89 (uses 1 / 2 / 4) vs Opus 5.5 $4,854-17,047; median 0.12-0.55 s per request vs 2.94-43.68 s. Hybrid (Jev judgments below 0.4 confidence go to an LLM): at most +0.04 agreement at 1.2-76× Jev's cost; routing only the safety rubric at a 0.2 cut raised caught unsafe answers from 31% to 49-51% | Cost and latency at frontier-judge agreement; LOST agreement to five judges (pediatric) and to GLM-5.3 and MiMo (HealthBench) | Preprint. Pediatric answers graded in English translation (on the Hebrew originals Jev's AC2 moved by at most 0.03); uses 3 and 4 mostly measure agreement with LLM graders; public sets may be in Jev's training (authors). Choice avoided because of the documented first-option lean (verified, Jev 1.13 jaggedness: known failure modes mode 8). Jev's yes/no confidence ran lower than its ordinal confidence, so one cut deferred more (58% at 0.4). Several authors also wrote the offload preprint in the row above |
| Medicine: 15 datasets, 8,742 instances in four task families (clinical judgment, safety review, physiological assessment, triage and navigation); meaning-preserving format changes, a missing-right-answer test and optional abstention (arXiv 2610.18529 v2, Yanrui Du and eight coauthors, Harbin Institute of Technology and Ant Group, 2026-10-16) | Jev 1.13 and 1.14 vs LLMs (Gemma4-31B-IT, Qwen3.8-27B, Qwen3.6-27B, Kimi-K2.6) and Jev-like models (D3-27B, StartLux-27B, OpenJev, Kev-4B, GLiNER2.5-Decide and others, none of them Jev); 23 configurations | Overall accuracy: Jev 61.6-62.3% (macro-F1 58.7-59.7%) vs D3-27B 66.54%, Gemma4-31B-IT 66.34%, StartLux-27B 64.83%. Clinical judgment 74.9-76.8% (Gemma 80.1, Qwen3.8 72.9); physiological assessment 35.9-39.4% vs Gemma 58.1; CTG-21 20.7-27.6%, below a 53.5% majority baseline; above 94% on ICD-proxy and MIMIC-CDM. Format changes moved Jev's accuracy by at most 0.45 points; decision reversals 5.69% (1.14) and 6.17% (1.13) vs Gemma 7.41%, Qwen3.8 10.75%, D3 4.74%. Swapping the right answer for none-of-these cost Jev 11.4-12.0 points; optional abstention (4.4-4.5% of items) raised retained accuracy only 0.3-0.5 points. Scoring each option as yes/no and normalising, instead of one Choice: at a 0.90 cut, 90.3-97.0% right vs 79.4-80.3%, but only 18.8-24.7% of items kept vs 39.9-42.3%; under Choice, 1.13 gave probability 1 to 1,819 decisions, 102 of them wrong | LOST overall to D3-27B and Gemma; near the top on format stability | Preprint; a research benchmark, not a basis for clinical use. Weak where numbers decide, as in Jev 1.13 jaggedness: known failure modes (verified in direction). Per-option yes/no scoring helped Jev but lowered D3's and StartLux's accuracy at matched coverage: test it per model (Failure reports: confidence misread and calibration) |
| Which exam questions medical students get wrong: each model rates the difficulty of 300 multiple-choice medical questions, compared with students' real first-try results (Spearman) (@0xnneo thread: cost, fine-tune, takeaways, 2026-10-17; chart transcribed; the authors' own data, not public) | Jev 1.14 and "Jev Router" vs Claude Opus, Sonnet and Haiku 5.5; Laya (not Jev) untuned and fine-tuned on 8,000 of the authors' questions; Gemini Argon on a 175-question subset | Opus 0.44, Jev 1.14 0.44, Jev Router 0.40, Sonnet 0.38, Haiku 0.33, Laya fine-tuned 0.23, untuned −0.04. On the subset: Gemini Argon 0.51, Opus 0.46, Jev 0.43. Jev 1.14 rated all 1,073 test questions for 2 cents in total, about 1/600 of Opus's price | Tie with Opus at far lower cost | Agreement is modest for every model: the best put only about half the questions in the right easy/medium/hard third, and real student data still beats them (authors). Laya fine-tuned is 0.23 on the chart, 0.30 in the thread. "Jev Router" is not explained (possibly OpenRouter's typesafe/jev-router, inferred). Jev 1.14 was jev-preview on these dates (Models, aliases, pricing, rate limits, context) |
Robots, browsers and reasoning suites
| Suite (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| Meta-World reach / push / pick-place, 2 seeds each; LIBERO drawer and microwave (embodied-jev, FBddcz) | Jev picks from code-built candidates (code does IK and safety) vs GPT-6 Astra | Meta-World, hierarchical (2026-09-21): Jev 5/6 = Astra 5/6 (both ran out of steps only on push-v3-0), estimated $0.018308 vs $3.605290. Direct four-channel control: Jev 2/6 vs GPT-6 5/6 (settings differ, not a clean ablation, our reading; per-arm budgets not in our capture; the earlier 40-step Jev pilots used a different budget and action interface). LIBERO (GPT-6 plans from dual-camera RGB-D, Jev picks the local moves): Jev acting was 81.7% cheaper ($0.39400 vs $2.15002) and 47.6% faster on the drawer; microwave: GPT-6 alone passed, the hybrid did not finish in 1,200 s (four runs in all: GPT-6 alone 2/2, hybrid 1/2) |
Cost, tie on hierarchy; LOST direct control | Development runs; LIBERO shown records include reruns. Its failures: Failure reports: where Jev broke, lost, or was the wrong tool |
| LIBERO microwave, drawer and alphabet soup, init state 0 (jev-libero, Dimweaker) | The simulator previews each of 27 inputs for up to 8 steps; only inputs meeting the task rules are offered; Jev picks intent, motion family, input | Microwave seeds 1 / 2 / 3: pass / fail / pass (14, 75, 53 decisions); seed 2 stopped at 28.33° when no candidate met the rules. Drawer pass (20 decisions, $0.001418). Alphabet soup failed first (kept lifting and lowering); after a config fix it passed containment, but release and settling were not shown. $0.012707 over 4 core runs | 3 of 4 core runs | Author: not a success rate. When code filters every option out, the run stops: plan a fallback |
| WebArena-Verified, a 12-task development subset of 812, official evaluator (Jevry, michaelswissa) | Jevry before and after fixes (an LLM plans, Jev picks actions) | 7/12 → 10/12 (an earlier run, speed-001, 9/12; tasks 47 and 102 still fail); median Jev request 443 → 392 ms, but median task time 23.367 → 44.169 s: faster decisions did not remove repeated navigation |
Accuracy up; slower | Repeatedly used subset, not a leaderboard result (author). A 449-option dropdown hit the 255-option Choice limit (HTTP API: POST /v1/systemone and GET /v1/models, Choice questions: verified); a 375,564-character request was refused as max_tokens_exceeded (code not in TypeSafe's docs: unverified) |
| ALFWorld text (12 episodes) and Blackjack (100 hands), 2026-09-19 (system1-agents, ThinkFlowLab) | Jev in the openJiuwen model slot vs a chat model (not recorded in the job folders) and a scripted reference (oracle plan; basic strategy) | ALFWorld 0.75 vs 0.917 (oracle plan 0.917), $0.0015 vs $0.0247 per episode (the author: 16 times the dollars). Blackjack: all three play the identical basic strategy (score −0.06 each), Jev at a fifth of the chat model's time per hand | Cost; Blackjack tie; LOST ALFWorld | Small n, 95% CIs overlap (0.50-1.00 vs 0.75-1.00); author's job folders. The README headline rests on single episodes: Head-to-head: Jev inside agents, routers and tool gates; Google Flights: Builds: browser, computer use and interface |
| 180 GPQA + AIME rows; 856 rows pooled (MetaCog, ItIsCuthNotCup) | An LLM writes reasoning paths, Jev Nouls judge each path and answer, more paths when unsure | 76.7% → 86.7% (180 rows); 75.0% → 79.4% pooled. Jev Noul ≥ 0.95 right 226/229; right pick 224/245 when paths disagreed; Jev 72/127 vs a local logprob judge 68/127 (p = 0.50); Brier 0.16 | Judge added accuracy | The LLM does the maths, Jev only judges (unverified). McNemar tests and a per-row ledger published |
| RACE-H (reading comprehension) and MMBench (images turned into dense captions), multiple choice (@Imroze_Aslam, 2026-09-29) | Jev vs LLMs and VLMs (not named in the text) | Within 1.3-2.4 accuracy points of models 10-40× its cost; "right on the Pareto frontier" on both | Cost; LOST accuracy narrowly | unverified: models, n and per-set numbers are in two charts, not transcribed; MMBench reached Jev only as captions |
| BIG-Bench Hard, 160 items over 4 tasks, 2026-09-20 (jev-skill, wuyoscar; CALIBRATION_RESULTS.md) | Jev vs DeepSeek V4.1 Flash (reasoning none), strict interface |
136/160 vs 120/160. confidence ≥ 0.90: 92/100 right, 8 wrong, 6 of them causal judgment (14/20 there). Top-probability bin [.9, 1]: 118 items, mean 0.9814, right 0.8898; middle bands not monotonic ([.70, .90) 70.4%, below .70 75.8%). Top probability ≥ .90 accepts 118 items, confidence ≥ .90 accepts 100. Model-panel rerun 2026-09-22 (README): 138 vs 108 |
Accuracy | The author calls the sample small and mixed: calibration unverified in general. Top probability and confidence are different signals (verified, Confidence vs probability): don't swap them silently. Agent pilot: Head-to-head: Jev inside agents, routers and tool gates |
| LLM Chess leaderboard, 80 games; Komodo Dragon levels 1-3, 10 games each (Maxim Saplin) | Jev picking from the code-built legal-move menu vs the board's LLMs and the engine | Elo 242.9 ± 117.5 (#59), beside o4-mini-medium 240.3 ($1.11 and 41.5 min per game); Jev $0.0015 and 35.6 s per game; every game completed. vs Dragon: 15 losses, 15 draws, 0 wins in 30 (5/5/0 per level). Our check 2026-09-24 of the unmerged feat/typesafe-jev-player branch's _logs/: 0 wins in 70 engine games (50 losses, 20 draws; levels 1-7; 20/20 lost at 6-7), 8 wins and 2 draws in 10 vs a random player |
Cost, latency; Elo tie with mid-pack reasoning models; LOST to the engine | Wide Elo interval. No Jev row on the live leaderboard on 2026-09-24 (branch unmerged): unverified against a capture; the 70-game tally is our own count (our 2026-09-24 vetting record, not published), not in a raw capture. Zero illegal moves came from the legal-move menu, not judgment. Failure reports: where Jev broke, lost, or was the wrong tool |
| Cognitive Reflection Test (150 new questions × 4 option orders), 210 random one-shot 3×3 matrix games, ALFWorld (140 seen, 134 unseen games), DeepMind Control Stacker (20 episodes, plus 5 new instructions × 8 seeds) (arXiv 2610.01834, "Code Owns the Simulation, Jev Owns the Evaluation", Yaodong Yang, Hongyao Tang, Yi Ma, Xingyu Fan, Weixun Wang, Jinpeng Li, Tianpei Yang; CUHK, Tianjin, Shanxi, Nanjing, CAIR-CAS; 2026-10-01; shared by @SciFi) | jev-1.13.0, one Choice per call, zero-shot; conditions plain / told to reason / prediction written into state by code; code lookahead vs Jev alone; a hand-written controller, random choice, a word-overlap rule |
CRT: right on 99% of lure questions (intuitive answer 1%; published open-form human answers took the lure 55%), 96% on trap-free controls. Games: perfect with a dominant action; 77–93% when the payoff-average shortcut agrees with the equilibrium, 28–37% when it misleads (lure 53–71%); a "reason first" instruction left that at chance; stating the opponent's move gave 97–99%. Asked separately Jev predicted the opponent right 90% of the time, yet chose right in 32% of calls with a right prediction vs 35% with a wrong one; predict, write into state, then decide: 33% → 94%. ALFWorld steps (seen / unseen): task-completing steps 93 / 96%, enabling steps where the shortcut is right 99 / 95%, where it misleads 36 / 43% (lure 56 / 51%). Next subgoal written in: 6% → 92% (goal-form tasks); code lookahead (each option shows the game's next output and newly possible commands) 36% → 79% at misleading steps. Whole unseen games: alone 42/134 (31%), with next output 79, plus new commands 116 (87%); a word-overlap rule with the same lookahead 18. Stacker: raw torques 0/20 (random 0/20); choosing among code-simulated skills 9/20 vs hand-written controller 8/20; 5 new instructions 22/40 vs 3/40 (random 0/40). ~94M input tokens, ~$4 |
Evaluation when code supplies the simulation; LOST when one call must simulate and decide | Preprint, one model version. CRT human and GPT figures are published open-form answers, reference only (authors). The ThinkFlowLab ALFWorld row above is a separate 12-episode run. $4 for ~94M tokens fits $0.042/M ($3.95, our arithmetic: verified, Models, aliases, pricing, rate limits, context). Repeat noise it reports: Request mechanics: billing, limits, latency, calibration and stability; lesson: Failure reports: wrong role, or options that don't fit |
Online-learning game: 4 actions × 64 rounds, 9 loss sequences fixed before play, every action's loss history in state (@ArdhenduTripat9 thread, 2026-10-05; tables transcribed; code in a Colab notebook, not captured) |
Jev Choice probabilities used as allocation weights vs follow-the-leader (FTL), Hedge, OMD, GPT-6 Luna (medium; one game per row); then a Noul per action ("will it take a loss next round?") turned into weights in code by argmin or softmin | Static regret, lower is better. Choice: lower than FTL on 7 of 9 games and best of all on two stable-winner games (2.36, 0.41), but 22.52 on noisy alternation and 21.94 when the past leader is penalised (best: OMD −1.49, Hedge 0.84). Noul forecasts: perfect alternation 2.60 → −27.00 argmin / −20.73 softmin (Luna −13.60); noisy alternation 22.52 → 3.50 / 1.08; past leader penalised 21.94 → 4.00 / 2.41; equal means 10.32 → 7.50 / 4.86 (Hedge 3.36). Softmin below Luna on 7 of 9 (our count) | Forecast-first beat Luna on most games; LOST as a direct Choice wherever the leader changes | Author: Choice weighting behaves like follow-the-leader; separating per-action prediction from allocation in code fixed most of it. Synthetic sequences, one game per row; unverified. Numeric sequences are a documented weak spot (Jev 1.13 jaggedness: known failure modes); weighing in code matches that advice |
| MMLU-Pro (12,032), ARC-Challenge (1,172), GPQA Diamond (196), HLE multiple-choice subset (494), MATH-500 as options (261) and as per-digit read-out (273), ARC-AGI-2 public set (167 grids / 120 tasks): 16,379 planned requests in three frozen suites, September 2026 (JevResearch/Jev-Research, report; README says revision 3, the page revision 5, both 2026-10-03; unaffiliated; secondary write-up by remio) | Service-reported jev-1.13.0, one shot, no retries, failures scored wrong ("all-requested"); vs 15 OpenRouter models run by the authors on the same frozen items (README says 12; MMLU-Pro on a 1,000-item stratified subset), vals.ai published scores (positioning only), Qwen3.5 9B (XNLI, political questions) |
MMLU-Pro 82.7% (CI 82.0–83.4; probability-weighted 74.0%), ARC-Challenge 97.9%, GPQA Diamond 76.5% (70.1–81.9; weighted 63.0%), HLE MC 21.9%, MATH-500 83.1% as options vs 13.6% digit by digit (0.817 vs 0.103 on the 224 shared items), ARC-AGI-2 0/120 exact tasks (per-cell 59.6–61.0%, a diagnostic encoding). Calibration ECE: MMLU-Pro 0.050, GPQA 0.088, HLE 0.415 (mean selected probability 0.633 at 21.9% right); gold answer at zero probability on 213 MMLU-Pro items. Shuffled options flipped 5.3% of MMLU-Pro answers, no net change. 19 contract-invalid responses in ~16k. Matched subset, MMLU-Pro (Jev 84.0% on those 1,000): GLM-5.3 88.0, DeepSeek V4 Flash 87.4, Qwen3.8 Max 87.2, Qwen3.8 Flash 86.3, GLM-5.3 Flash 85.4, MiMo v2.6 Pro 85.2, Qwen3.7 Flash 84.2, MiMo v2.6 Flash 84.0, gpt-oss-120b 81.9, gpt-oss-20b 74.7, Mistral Small 3.2 57.7, Mistral Nemo 36.9, Granite 4.0 H Micro 32.9, Llama 3.1 8B 31.9, Gemma 3 4B 31.4; GPQA: 7 of 15 above Jev (best GLM-5.3 87.8); HLE: 4 of 12 above (best MiMo v2.6 Flash 34.8). Provider-reported cost of those 1,000 MMLU-Pro items $0.0064 (Llama 3.1 8B) to $10.76 (GLM-5.3); Jev's whole 12,032-item run $0.28. XNLI, 1,200 premises in 6 languages: Jev beat Qwen3.5 9B (no reasoning) in all six (English 87.5 vs 83.3%; others 77.4–82.4 vs 71.3–76.4%). Politically sensitive closed-book questions: Jev did not repeat Qwen's PRC-line denials (Taiwan's administering government: Taipei 20/20 vs Beijing 20/20) | Knowledge multiple choice at a tiny cost; LOST multi-step (HLE, ARC-AGI-2, digit read-out) and to 7 matched cheap models on the MMLU-Pro subset and GPQA | unverified: aggregates published, raw responses and licensed items not, so re-rendering the page is not re-scoring (authors). Published comparison scores mostly use reasoning and few-shot prompts. GPQA differs from the 78.3 Cloudflare reported for Jev and MMLU-Pro coincides (Rival decision models: Clef, Liquid d1, GLiDE, Drex and other non-Jev System One models, separate runs); Archer Hume's MMLU-Pro 84.6% is another sample (Request mechanics: billing, limits, latency, calibration and stability). Baselines ran at provider defaults, except five cells kept from an earlier temperature-0, low-effort protocol (all four Granite cells, Qwen3.8 Max on MMLU-Pro), with 1,157 answer-parsing recoveries. $0.28 / 12,032 ≈ 550 input tokens per question at $0.042/M (our arithmetic), plausible. Knowledge read strongest on 2024-era facts; size and architecture are their inference. Latency, rounding, option order, repeat noise and confidence: Request mechanics: billing, limits, latency, calibration and stability; small-model reading: Head-to-head: Jev vs small, fast and local models |
| ThinkFast no-CoT suite (14 benchmarks), nocot-bench's NCRI ladder (279 models), three transcript-monitoring sets: coding sabotage, SHADE-Arena action-only, sandbagging (LessWrong, Dewi Gould and James Mann, 2026-10-07; summarised by @jevwatch) | Jev 1.13 vs GPT-5.5, Opus 4.7 and Opus 4.1 answering with no chain of thought. Monitors: the benchmark's instructions plus one Score whose levels come from its rubric, suspicion = probability-weighted level. Multi-part answers built from one question per part (a sudoku = 81 Choices). NCRI's free-form questions turned into option lists (up to 255 consecutive integers) | ThinkFast mean, chance-corrected: Jev 46.8% vs GPT-5.5 58.1%, Opus 4.7 53.7%, Opus 4.1 40.4%. GPQA-Diamond 65.2 vs GPT-5.5 51.8; within 5 points on N-hop lookup, CyBash, training-poisoning; behind on BEA-24, intuitive physics, Sally-Anne; weakest on multi-part answers (sudoku, test-case pass/fail, chess, Tower of London). Monitor AUROC: coding sabotage 0.99 (GPT-5.5 and Opus 4.7 0.99, Opus 4.1 0.74), SHADE-Arena 0.90 (0.93, 0.92, 0.86), sandbagging 0.94 (0.91, 0.97, 0.86); caught at a 1.5% false-positive rate: 89% vs 93–94%, 47% vs 67–71%, 55% vs 68–76%. NCRI 49.7 [45.2, 53.9], rank 209 of 279, beside GPT-3.5-turbo-instruct and GPT-4.1-nano (rank 38, 108, 154 with 5, 20, 50 integer options). Cost about 1/200 of Opus 4.7's | GPQA-Diamond; monitor AUROC near the frontier at a fraction of the cost; LOST catches at a strict false-alarm budget, multi-part answers and multi-step reasoning | Authors: Jev is "very jagged"; cheap enough for routing and always-on agent monitoring, but it leaves no reasoning trace to audit. NCRI rank is an upper bound (picking from a list is easier, and the LLMs answered free-form). Jev read only transcripts within its limit (134/511 SHADE, 297/379 coding, 244/408 sandbagging; all models scored on that subset): 32k per question verified (Models, aliases, pricing, rate limits, context); 255 options verified (HTTP API: POST /v1/systemone and GET /v1/models). The post's summary gives other 1.5%-rate figures than its monitor section (88.0 vs 90.8, 45.9 vs 59.3, 41.3 vs 63.5); @jevwatch's "46% vs 59%" is the summary's. Its parameter guess (fit at about 17B) is, by its own account, rough at best |
| 13 multiple-choice benchmarks: knowledge (MMLU-Pro, MMLU-Redux, MMLU 14,042, GPQA Diamond 198, C-Eval, HellaSwag, WinoGrande, TruthfulQA 817), reasoning (ARC-Challenge 1,172, MathQA, AQuA-RAT 254), multilingual (MMLU-ProX, MMMLU) (arXiv 2610.11978, Xing Li, Qingcheng Chang, Jinzhong Ning, Changfeng Xu, Shenlong Zhang, Yijia Zhang, Ling Luo, Hongfei Lin; Dalian University of Technology; v1 2026-10-08) | Jev (one Choice per item) vs 19 LLMs: six frontier (GPT-5.6 Sol, Claude Opus 5.5, Gemini 3.8 Flash, Qwen3.8-Flash and -Max, DeepSeek-V4.1-Flash), seven representative, six small (< 10B) | Best of all 20 on MMLU-Redux 95.0 and ARC-Challenge 97.6; vs the frontier median TruthfulQA +4.5, WinoGrande −0.3, MMLU −1.2, MMLU-Pro −5.8, C-Eval −8.3, GPQA −8.9, AQuA-RAT −7.9, MathQA 75.0 vs 92.7 (−17.7), below all 19. Outside maths ahead of the representative tier in 68 of 76 comparisons and of the small tier in all 62 | Knowledge and commonsense; LOST maths word problems to every model, Chinese-exam knowledge (C-Eval) | Authors' numbers, preprint. Jev and frontier models one run each; 104 of 169 smaller-model scores are published figures with other prompts (authors: rough comparison only). The maths gap fits documented failure mode 2, keep arithmetic in code (verified, Jev 1.13 jaggedness: known failure modes). Close to arXiv 2609.37647 (Bonn suite above) on MMLU and WinoGrande (≤ 0.5 points) and C-Eval (1.4); HellaSwag 6.2 lower, likely the option template (authors; failure mode 8) |
| MedQA (USMLE, 4 options), all 1,273 test questions, 1 attempt each, 14 decision models through OpenRouter, every request within 11 minutes from one residential IP in Tokyo (@zarate_ai, 2026-10-11; 4 charts transcribed) | Jev 1.13 vs 13 rival decision models (none is Jev: Rival decision models: Clef, Liquid d1, GLiDE, Drex and other non-Jev System One models): Perplexity Decider v1.1 27B, Inception Mercury Decide, LiquidAI d1, OpenAI GPT-6 Luna Decisions, Microsoft Decision-1, Cloudflare Clef, Clef Omni and Clef Flash, Nace.AI Drex v1.5, Together Tev1 4B Experimental, Upstage Solar Decide and Decide Flash, Kev 4B | Accuracy: Decider 88.1%, Jev 87.7% = Luna Decisions, d1 87.0%, Mercury 85.2%, Clef 84.9%, the rest 67.1-83.7%. Cost of all 1,273 (median billed cost per question × 1,273): Mercury $0.0076, Decider $0.0078, Jev $0.0293, Luna $0.0412, Clef $0.1124. Median latency: Clef Omni 0.70 s, Jev 0.71 s, Luna 0.74 s, the rest 0.81-1.47 s. Calibration: Jev ECE 0.01 (lowest, with Clef), AUROC 0.90 (d1 0.91). The author's rank points: Mercury 35, Decider 34, Jev 32 (3rd of 14) | Latency, calibration; LOST cost to Mercury and Decider, accuracy by 0.4 pts to Decider | One run; the top models' accuracy intervals overlap (author); public set, so pretraining overlap is possible; the points scheme is the author's; Luna Decisions refused 3 (counted wrong); latency is the client's round trip through OpenRouter. $0.0293 implies ~550 input tokens per question at $0.042/M (our arithmetic; plausible, token counts not published) |
| ALFWorld text (134 valid-unseen tasks) and AndroidWorld (116 task types, text UI), 5 agents (ReAct, Reflexion; T3A, AutoDroid, FastAgent), 3 runs each, 4,500 executions (arXiv 2610.13092, "When Faster Decisions Do Not Make Better Agents", Yuchen Ling, Shengcheng Yu, Zhenyu Chen, Chunrong Fang; Nanjing University, TU Munich; v1 2026-10-09; via @ValesyaAI) | Each agent's own gpt-5.6-luna action picker vs jev-1.13.0 picking from candidates, one Jev request per step, Luna kept for open text and reflection; on ALFWorld also Luna given Jev's candidate list |
Success: ALFWorld ReAct 96.02% (−1.49 pts; −1.00 vs Luna on the same list), Reflexion 98.01% (−1.74; 0.00); AndroidWorld T3A −13.22, AutoDroid −2.30, FastAgent −18.10 pts (T3A and FastAgent intervals wholly below zero). Per decision: cost −66.1 to −78.4%, time −73.9 to −81.5% (4.13-8.13 s → 1.04-1.75 s), tokens +53.5 to +134.9%. Per task: cost −17.1 to −68.8%; time −47.6% (ReAct) to +0.7% (FastAgent). Cost per success fell for 4 agents, rose 7.7% for T3A (68.70 → 73.97 mUSD; interval spans zero); FastAgent time per success 239.4 → 377.8 s. Reflexion: first trial 383 vs 376 of 402 right, but recovered 11 of 19 failures vs 25 of 26 | Decision cost and speed; LOST success on two mobile agents | Preprint; one seed on fixed tasks; text-only inputs to suit Jev. Traces: failed recovery and "done" with the goal unmet ended runs. Not the same work as ThinkFlowLab's 12-episode row or arXiv 2610.01834 above. Authors: judge by cost per completed task, not per decision; agent A/Bs: Head-to-head: Jev inside agents, routers and tool gates |
| Reinforcement learning: nine MiniGrid tasks and three Atari games (Seaquest, Freeway, SpaceInvaders), DQN students, three training seeds (arXiv 2610.11692, Yi Ma, Tianpei Yang, Yaodong Yang, Weixun Wang, Hongyao Tang; @Trtd6Trtd, 2026-10-13) | Frozen Jev as a zero-shot policy vs Qwen3-8B (non-thinking, same structured input); then as reference policy, exploration judge and replay rater vs plain DQN and prioritized replay | Direct control: Jev matched or beat Qwen3-8B on all nine MiniGrid tasks; Atari mixed. Jev's success with vs without engineered navigation hints: FourRooms 96 → 40%, MultiRoom-N4 100 → 38%, DoorKey-8×8 100 → 100%. Training with Jev beat plain DQN on several tasks, including some where DQN alone made no progress; no configuration was best everywhere, and Jev-rated replay did not consistently beat prioritized replay (most results are figures, not captured as numbers) | As a prior and guide; LOST as a long-horizon controller on its own | Jev's inputs carried information the student lacked (structured descriptions, task rules, route features). The authors rule out the value-function role because its update does arithmetic on the answer, which fits failure mode 2 (Jev 1.13 jaggedness: known failure modes). Three seeds |
| System1-Arena: 92k questions across 105 tasks from 268 public sources (classification and knowledge, applied decisions, evaluation and control, agent and action) plus 138 closed-loop environments (web shopping, SWE-bench coding control, robotics, games, driving; some in real time) (arXiv 2610.21244, Tianyu Chen, Xinyang Liu, Ruyi An, Zhendong Wang, Mingyuan Zhou, UT Austin, v1 2026-10-15; code, leaderboard; relayed by @neuralloot) | 30 models on the same decision requests: 7 closed LLMs (verbalized probabilities), 13 open LLMs read at the answer token, 10 System-1 models incl. Jev 1.13, Microsoft-Decision-1, the OpenAI Decisions API and open Decision-2.0 models (not Jev) | Table 1 (78,362 questions over 87 tasks): Claude Opus 5.5 most accurate, 81.1% at $10.38 per 1,000 requests and 2,293 ms; Jev 1.13 72.6% (8th of 30), Brier 0.349, 321 ms, $0.052 per 1,000, so Opus costs about 200× more (our arithmetic); Gemma-4-31B read at the answer token 73.9%, Decision-2.0-Vega-27B 73.0%, GPT-6 Luna 71.2% at $0.14, Microsoft-Decision-1 70.9% at $0.038, OpenAI Decisions API 68.6%. Agent steps (3,481) and closed loops (127 environments): Jev step accuracy 67.8% with no answer on 13.2% of steps, loop success 17.4% vs Vega-27B 26.5%; best loop success 32.9% (a large open LLM) | LOST to Opus and other frontier LLMs; first among hosted System-1 APIs | Preprint. Labels inherited from public, partly old sources (possible training overlap); hosted latencies include the network; one draw per request. The abstract's 92k / 105 / 138 describe the whole benchmark, the tables 78,362 / 87 and 127 (as @neuralloot quotes) |
| Cognitive and social biases: 41,500 responses over cognitive-reflection tasks, five classic bias paradigms and a pricing test for ethnic and gender bias, then 15,000 more over 30 conjunction tasks (arXiv 2610.19619, Ilaria Imperatrice, Taisia Kameneva, Adam Eric Greenberg, Alessandro Romano, Bocconi University, v1 2026-10-14) | Jev 1.13 against the human patterns each paradigm is known for | Chosen answers mostly avoid the human bias patterns; cognitive-reflection problems answered very well; small demographic gaps in the pricing scenario. But answers moved with irrelevant numbers and with option order, and bias manipulations shifted returned probabilities even when the pick held. A well-known probability problem: right in every response; an adapted version: the same rule broken in every response. Across 30 conjunction tasks the rule was broken in 54% of responses, accuracy swinging with wording and answer order | Picks mostly unbiased; LOST on variants and probabilities | Preprint; numbers from the abstract. Option order and number sensitivity match Jev 1.13 jaggedness: known failure modes (verified in direction); test your own variants, not only famous items |
| RealtimeBench: 180 tasks over six game types (Mario, Snake, Tetris, Temple, River, Bird); the game keeps running while the agent decides; text observations and harness-built legal actions (arXiv 2610.20665, "The World Won't Wait", Ningxin Pan and five coauthors, Peking University and Samsung Research Beijing, v1 2026-10-14) | Jev (version not stated) vs GPT-6.1-sol (medium reasoning, ~19 s per response), other LLMs and decision models (Clef 27B and Clef-flash, StartLux 4B and 27B, GLiClass, replicas; none of them Jev) | Paused game: GPT-6.1-sol ahead in five of six game types. Live clock, mean progress 0-100: Jev 13.51 vs GPT-6.1-sol 1.51, but StartLux 27B 16.62 and Clef 27B 16.34 beat Jev; Jev 0.52 s per response, $0.0111 per task. TORSI (a slow teacher writes reusable instructions between attempts, Jev acts with no teacher calls) raised Jev from 13.51 to 24.19 in two rounds, in all six game types | Beat the reasoning LLM live; LOST to two rival decision models | Preprint; ten test tasks per game type, one run each mostly. Other game comparisons: Head-to-head: games, simulators and real-time control |
| Realtime-Bench (not RealtimeBench above): 50 tasks, 30 web (live auctions, room holds, a market-dip purchase) and 20 games, where the world advances by each model's real latency (Wall Clock); 3 runs (arXiv 2610.22155 v3, Jingyu Tang and nine coauthors, UT Austin, Ohio State, UIUC, NVIDIA, Northwestern, Steadyworks, v1 2026-10-15; @JingyuTang621 thread, 2026-10-23; chart transcribed) | Jev 1.13 and OpenAI Decisions via OpenRouter, one Choice over harness-built GUI actions (Jev reads an accessibility-tree text view; OpenAI Decisions also gets screenshots), vs GPT-6-Astra, GPT-5.6 Sol, Claude Fable 5.1, Claude Opus 5, Gemini 3.8 Flash, DeepSeek V4.1 Flash, Kimi K3 and three real-time streaming models (GPT Realtime 2, Gemini 3.1 Flash Live, Gemini 3.8 Live) | Pass^3 (all three runs succeed): GPT-6-Astra (high) 36% at 3.99 s per decision; Gemini 3.8 Flash 14-22%; Fable 5.1 and Opus 5 (low) 10%; Jev 8% at 0.47 s, the fastest; GPT Realtime 2 6%; OpenAI Decisions 4% at 1.13 s; the rest 2-6%. Jev's runs 16 / 10 / 10% vs Astra 42 / 42 / 44% | Latency; LOST success to Astra, Gemini 3.8 Flash and the Claude models; ahead of OpenAI Decisions | Jev ran only the 30 web tasks (its text view misses game state), yet its rates are multiples of 2%, so they appear to be over all 50 with the 20 games as misses (inferred): read 8% as a floor for web-only use. Both decision models ran on the wall clock only. A failure example: Jev bought after the price had rebounded past the limit, on a 0.4 s call. Preprint |
| AgentDecisionBench: decision models from isolated questions up to control of a running agent, on public benchmarks and the authors' DecisionGym; 500 tasks for the harness test (arXiv 2610.21412, "JevAgentHarness", Kunlun Zhu and seven coauthors, UIUC and others, v1 2026-10-15; relayed by @dair_ai; models and harness promised at github.com/JevAgentHarness) | Jev 1.13 (pinned typesafe/jev-1.13) and other decision models taking the LLM's routine choices (tool, arguments) vs the LLM deciding everything; AgentDecision-4B, an open 4B model trained on states the agent reaches after its own mistakes with Jev's answers as labels (not Jev) |
The decision model most accurate on isolated questions solved only 69.5% of agent tasks; delegating tool choices without a fitted confidence threshold cost 19.0 points of success (a fixed 0.5 threshold: −21.5). Thresholds fitted per choice on the agent's own trajectories let tool and argument cascades cut cost 14.2% without changing the outcome of any of 500 tasks. On tasks unlike the training data the agent solved 91.5% with AgentDecision-4B vs 82.0% with Jev; on familiar tasks such 4B models made agents 81.1-89.0% cheaper and up to 2.5× faster, within 3 points of success | LOST to a distilled 4B out of distribution; Jev-gated harness level on outcome at lower cost | Preprint. Below the threshold the LLM decides; a fixed threshold is the main failure. Gates: Patterns: agent internals, routing, gates, context and memory; other agent A/B tests: Head-to-head: Jev inside agents, routers and tool gates |
| Multi-agent workflows: routing and review take 31.7-43.5% of generative-LLM tokens in three profiled frameworks; four benchmarks, two LLM backbones (arXiv 2610.21411 v2, "Jev-LiteMAS", Kaixiang Wang, Zhaijiacheng Zhou, Jiong Lou, Jie Li, 2026-10-16) | Jev picks the next expert from the task and earlier experts' outputs and screens intermediate artifacts (passing ones go on, flagged ones get an LLM re-review) vs the LLM-only baseline (LAS) | Generative-LLM tokens down 12.7-54.6%, accuracy within 1.34 points, lower latency in seven of eight settings; grounding routing in the experts' actual outputs avoided up to 18.0 points of accuracy loss | Cost at near-equal accuracy | Preprint; abstract numbers; Jev version not stated. LLMs still plan and write repair feedback; Jev only decides |
NARCBench multi-agent collusion, Jev black-box monitor vs vote counting and white-box probes (jev-collusion-monitor): Builds: coding agents, harnesses and orchestration.
How to read these
The reading notes on Head-to-head benchmarks: Jev on public datasets and suites apply here too (public is not unseen, option order moves answers, thresholds are per dataset, check simple baselines). Numbers belong in code (Jev 1.13 jaggedness: known failure modes). Patterns: Patterns: agent internals, routing, gates, context and memory P04 loop control, Patterns: judging, search, documents, real-time and markets P24 real-time; Patterns: gates, simulation, personas and other shapes (P38+) P41 engine owns the truth.
Related
- Head-to-head benchmarks: Jev on public datasets and suites: text classification, intent and legal sets; Head-to-head: Jev inside agents, routers and tool gates: agent A/Bs; Head-to-head: games, simulators and real-time control: games; Eval boards: JevBench by version, Jevals.com, typed-decisions and other independent suites: boards; Cost ledger: published cost per Jev decision
- Confidence vs probability, Testing and evaluating a Jev workflow: run your own comparison
Sources
Links inline; raw captures listed in the frontmatter. The llm_chess 70-game tally comes from our own 2026-09-24 vetting of the harness branch (our 2026-09-24 vetting record, not published), not a capture.