Agents: read the raw Markdown of this page, or start at llms.txt.
~/wiki/ideas
Head-to-head: Jev inside agents, routers and tool gates
TL;DR When builders A/B-tested a whole agent with and without Jev, the reliable gains were smaller context and lower cost. Task success usually stayed the same, sometimes dropped, and wall time often rose: an agent ignored an optional Jev tool (0/150), a cheap model alone matched a Jev router, a Claude subagent picked skills better, and gates over-denied without a written policy. Measure the whole agent, not the decision. Split from Head-to-head: Jev against other models and methods on 2026-09-25.
Numbers as each builder published them, on jev-1.13.0, mostly small n and one run. LOST = the Jev arm lost on the stated goal. Patterns: Patterns: agent internals, routing, gates, context and memory P02 model and effort routing, P03 safety gates, P05 worker routing; Patterns: browser, computer use, voice and product UI P12 browser. Rows moved here from Head-to-head: Jev against other models and methods: LiteLLM, JevRouter, langchain-skill-router.
Routing a request to a model
| Task (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| Auto Router tier, 80 author-labelled prompts × 3 (LiteLLM blog, Moe Khalil; vendor: LiteLLM ships the router) | Jev vs Claude Haiku 4.5 | Tier match 95.00% vs 73.75%; p50 126.81 vs 688.40 ms; classifier cost 96.1% lower ($0.0077 vs $0.1985 for 240 calls) | All | Labels and prompts by the same author, not reviewed; Jev 87.5% on long cases vs 97.5% short; final answers not graded |
| Model tier for 120 prompts, blind-scored by two judges (tiershift, iamvatsalpatel) | Jev router (11 questions in one call) vs always GPT-5.6 Sol, always DeepSeek Flash | Quality 4.77 at $7.95 per 1,000 prompts vs Sol 4.77 at $13.23 vs Flash 4.78 at $0.37. Routing p50 180 ms, $0.0418 per 1,000; all 18 high-stakes prompts went to the mid tier | LOST: the cheap model alone matched, so routing added cost | Author's own reading; single-turn prompts. npm and PyPI ship 0.1.0, the benchmarked policy is repo 0.2.0 |
| Pi coding-agent router, 5 tasks, 2026-09-20 (pi-jev-router, philippdubach) | One Jev request (5 questions) feeds a quality/cost/latency pick over the OpenRouter catalogue, vs fixed Sonnet 5 | All arms 5/5; cost per success $0.0399 vs $0.1144 | Cost | Jev's own cost excluded; the author says 5 tasks prove no general saving. OpenRouter base URL and model ID verified. npm pi-jev-router is another publisher's |
| Prompt router in a Claude Code plugin (claude-jev, 0x7067) | Jev predicts the task type from the prompt, against labels derived from transcripts | 34.6% on 1,613 prompts; humans agreed with the derived labels only 52.5% on 120; a coarse 3-way taxonomy scored 58.8%, below a constant guess (64.7%) | LOST | Same plugin's rule checker per edit: 22 of 247 accepted real edits blocked (17 from one repo's no-comments rule), 17/29 planted violations caught, 0/24 near-misses blocked, median 0.80 s. Compaction ~0.9-1.1 s vs ~117 s; the planted constraint survived all 56 sessions |
Picking skills and tools
| Task (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| Skill routing in a bank-statement agent, 236 skills, 55 conversations × 5 turns (langchain-skill-router, deyna256) | Jev picks the skills vs the full catalog in the agent's prompt | 25.8k vs 113.0k input tokens per turn (4.4× less); right skill in front of the model 85% vs 55%; correct answers 90% vs 88% (+1.5, 95% CI −1.5 to +4.7, p = 0.63) | Context size; accuracy tie | Generated data, one agent model. A wrong skill loaded: 72% correct vs 92% with the right one. An earlier run was 5 pts below the full catalog (six contradictory skills) |
| First-5 tool-call prediction, 10 Toolathlon tasks (JevRouter, BillionsBobby) | vs DeepSeek V4.1 Flash | Position-wise hits 38% (serial) / 44% (decompose + thread) vs 24%; 1.58 s (serial) vs 8.65 s per task; $0.0058 vs ≈ $0.0407 per 10 tasks | All | Author-reported, n = 10; measures routing prediction, not task completion |
| 162 routing fixtures for roles, skills and tools (agent-dispatcher, nahid-sparktales) | Jev as judge vs a Claude subagent vs keywords | Top-1 143/162 vs 158/162 vs 23/162; skill precision/recall 0.68/0.71 vs 0.74/0.90; tools a tie; Jev median 358 ms; five reruns 136-143 | LOST to the Claude subagent, so the author ships Jev off by default | The Claude column is a reconstruction; only 20 tool cases. The repo has since become a retrieval engine |
| 72 synthetic requests over 24 synthetic skills (jev-agent-skill-router, GodsBoy) | Batched Choice rounds, then need / ambiguity / fit Nouls gate in code, vs a lexical baseline | 68/72 vs 51/72; 0 wrong routes, 18 sent to review, 1 invalid response; median 1,287 ms, p95 1,406 ms; 255,897 input tokens (≥ $0.0107) | Accuracy | Questions revised after an earlier run (64/72). The author reports the cookbook pin jev-1.12 returned HTTP 400 on 2026-09-16: unverified; Models, aliases, pricing, rate limits, context lists jev-1.13.0 |
| Capability-doc tree search, 20 scenarios × 4 setups (JCR, NiazMorshed2007) | Jev ranks each node's children (with a no-match option) vs the agent reading all docs | Agent input −85% with Opus 5, −23% with GPT-5.6 Sol; cost −67% and −16%. Sol slower with JCR in 19 of 20 scenarios (median 23.2 → 45.4 s) | Context, cost; LOST time | One run each, lookup only; only a chart is committed |
| Hermes skill suggestions, 292 skills (typesafe-skill-router, DECRUX9812) | Port of the Cookbook: Skill suggestion cookbook (wide Choice with none_of_these, 3 gate Nouls) |
~0.6-1.2 s and ~$0.001 per turn; a second check (n = 12) p50 1.93 s, max 8.56 s; fits ~0.14 lower in Spanish; first 4 live turns: one miss, one off-target |
Unclear | Its "budget for output tokens" contradicts docs (output is free, Models, aliases, pricing, rate limits, context) |
| Tool steering in coding agents, 120 sessions: 6 models × 2 chess tasks × 5 runs (jev-gateway, vinilana; companion jev-gateway-bench) | A Jev Choice over the agent's tools forces or hints a tool when confident, vs no routing | Bug fix: cheaper for every model (GPT-6 Astra and GPT-5.6 Sol −57% output tokens). Feature task: Opus 5 +61% input, +83% time; Sonnet 5 also worse; GPT-5.6 Luna solved 3/5 vs 5/5 without routing | Cost on bug fixes; LOST on the feature task | Adds a Jev call per turn (~0.5-1 s in our 2026-09-25 sweep note; not in the captured READMEs, which show one 712 ms key check: unverified); fails open. Author confirmed OpenCode Zen free and paid IDs live on 2026-09-24 |
| Judgment tools offered to an agent, 150 Pi tasks and a 3 × 3 outcome study (jev-judge-mcp, PyModel) | Jev as an optional or forced MCP tool | Optional: never called (0/150). Forced: +10.4 s median per task; Jev itself 465 ms median. Outcomes: Claude Code 6/9 in both arms, 4.6 s slower with Jev; Pi 6/8 in both, 85.8 s slower | LOST time; no accuracy gain | Paid, small samples; Jev's accuracy not measured |
Approval and safety gates
| Task (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| 156 real mined shell commands, 2026-09-17 (hermes-jev-approvals, anpicasso) | Six questions in one request, combined by code rules, vs Hermes's small chat-LLM reviewer | 10 human prompts vs 42; 405 ms vs 3,968 ms average. Without an operator policy Jev denied 17 of 153, 8 of them routine Chrome kills, and denied or escalated plain git push origin dev until a policy clause was added (--force to main stays denied) |
Fewer prompts, latency | Thresholds tuned on the same data; no production use |
| Same plugin, independent assay: 28 unique synthetic commands, 156 observations (Bear Huddleston) | Jev plugin vs GPT-5.4 mini vs GPT-5.6 Luna reviewers | Mean reviewer time 1.062 / 1.319 / 1.198 s (Mini/Jev 1.24x, not the author's 9.8x). The plugin sent a 2,000-character prefix of a 2,909-character policy and approved a deletion the cut exception forbade; Mini and Luna denied it. Luna matched all 38 reviewed primary verdicts | Latency; LOST where input was cut | Already-seen cases, no holdout; cost is a published-rate estimate (Jev $0.00229 per 44 calls) |
| DeepSeek Harness tool gate (dsh-jev, buberlo) | Jev narrows tools and checks each call, vs a rule written in the prompt | Author: no better than the prompt rule when the model complies (0/10 deletions), and slower. A weaker model tried deleting 10/10 times; Jev denied all 31 attempts. k3s replay: denied the broad allow-all patch (0.960) but also falsely denied a correct reset (0.940) | Tie; blocks a non-compliant model | One machine, one model, one day. npm 0.1.2 and 0.1.3 are broken |
| Coding-agent guard with a rules table first (jev-kit, jonathanavis96) | One Jev question for the ambiguous rest; deny needs confidence ≥ 0.8 and margin ≥ 0.4 | General-risk rule 20/21; tier guard 56/57 (ECE 1.9% on 50 rows); search-intent guard 84.4% (ECE 12.6% on 32 rows). The 30-session A/B produced zero denies, so no savings shown. Browser clicks 9/9 at 314-486 ms median vs Sonnet 9/9 at 1.1-1.5 s vs Haiku 4/9 (n = 3 per cell). Wikipedia hops: 15/18 when named, 0/6 when not | Latency on clicks | Invasive install (hook on every tool, systemd timers). Author numbers, 2026-09-19 and 09-23 |
Advice and decisions inside agent loops
| Task (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| Pre-declared pilot, 24 episodes, DeepSeek V4.1 Flash, advice at turns 3/6/9 (jev-skill, wuyoscar) | Agent with vs without Jev advice | Completed 12/12 vs 10/12 (0 Jev wins, 10 ties, 2 losses); cost $0.0051 → $0.0091; API time per episode 6.33 → 12.57 s. A 2026-09-22 rerun (4 pairs): 3/4 → 4/4 | LOST the pilot | Author: neither run proves a general benefit. Every request and response committed. Its BBH calibration: Failure reports: confidence misread and calibration |
| openJiuwen agents with a decision model in the model slot (system1-agents, ThinkFlowLab) | Jev vs an unnamed chat model | Series of 2026-09-19: Blackjack 100 hands tie; ALFWorld 12 episodes 0.75 vs 0.917 at 1/16 the cost; 2048 (20 moves) 64 vs 80; flight booking 3/3 in every arm | Cost; LOST ALFWorld and 2048 | The README's "6× faster, 25× cheaper at the same score" rests on single episodes |
How to read these
- Measure the agent, not the call. Jev's own latency (180-465 ms here) was never the problem; forced calls, extra turns and repeated navigation added seconds (jev-judge-mcp, JCR, wuyoscar). Browser agents show the same: Head-to-head benchmarks: Jev on public datasets and suites (Jevry).
- Gains: less context in front of the model (langchain-skill-router, JCR), cheaper bug fixes (jev-gateway), fewer human prompts (hermes-jev-approvals), cheaper routing classification (LiteLLM).
- Losses: a cheap model alone (tiershift), a Claude subagent (agent-dispatcher), a prompt rule (dsh-jev), no Jev at all (wuyoscar pilot, jev-gateway feature task). Effects depend on the model: routing helped GPT-5.6 Sol and hurt Opus 5.
- Gates need a written policy passed whole: without one Jev over-denied (hermes), and a truncated policy flipped a verdict (Bear Huddleston). Deterministic rules first (Patterns: agent internals, routing, gates, context and memory P03).
- Agents ignore optional tools: if Jev must run, call it from code, not as an offered tool (jev-judge-mcp).
Related
- Head-to-head: Jev against other models and methods: app-level comparisons and cascades; Head-to-head benchmarks: Jev on public datasets and suites: public sets; Cost ledger: published cost per Jev decision: cost per decision
- Failure reports: where Jev broke, lost, or was the wrong tool: what broke; Builds: coding agents, harnesses and orchestration, Tools and integrations hub: MCP servers, plugins and connectors: the tools themselves
- Jev with coding agents: not a drop-in for the LLM behind Claude Code, Cursor, Copilot: what TypeSafe says about coding agents
Sources
Links inline; raw captures listed in the frontmatter (private repo).