---
title: "Head-to-head: Jev inside agents, routers and tool gates"
type: community
source_tier: community
tags: [head-to-head, agents, routing, tool-gates, community]
created: 2026-09-25
updated: 2026-09-25
confidence: medium
sources:
  - raw/community/docs-litellm-ai-blog-jev-auto-router-benchmark.md
  - raw/x-repos/iamvatsalpatel__tiershift.md
  - raw/x-repos/iamvatsalpatel__tiershift__bench-results.md
  - raw/x-repos/philippdubach__pi-jev-router.md
  - raw/x-repos/philippdubach__pi-jev-router__eval-results-benchmark-report-2026-09-20t18-41-53-381z.md
  - raw/x-repos/0x7067__claude-jev.md
  - raw/x-repos/BillionsBobby__JevRouter.md
  - raw/x-repos/deyna256__langchain-skill-router.md
  - raw/x-repos/nahid-sparktales__agent-dispatcher.md
  - raw/x-repos/nahid-sparktales__agent-dispatcher__docs-jev.md
  - raw/x-repos/GodsBoy__jev-agent-skill-router.md
  - raw/x-repos/GodsBoy__jev-agent-skill-router__results-final-v1-1.md
  - raw/x-repos/NiazMorshed2007__jcr.md
  - raw/x-repos/DECRUX9812__typesafe-skill-router.md
  - raw/x-repos/vinilana__jev-gateway.md
  - raw/x-repos/vinilana__jev-gateway-bench.md
  - docs/sweep/2026-09-25-mrjev/partc.md
  - raw/x-repos/PyModel__jev-judge-mcp.md
  - raw/x-repos/PyModel__jev-judge-mcp__evals-reports-bench150.md
  - raw/x-repos/PyModel__jev-judge-mcp__evals-reports-agent-outcomes.md
  - raw/x-repos/anpicasso__hermes-jev-approvals.md
  - raw/x-repos/anpicasso__hermes-jev-approvals__docs-metrics.md
  - raw/community/bearhuddleston-dev-reports-jev-approvals-live-sandbox.md
  - raw/x-repos/buberlo__dsh-jev.md
  - raw/x-repos/buberlo__dsh-jev__docs-benchmark.md
  - raw/x-repos/jonathanavis96__jev-kit.md
  - raw/x-repos/jonathanavis96__jev-kit__docs-measurements.md
  - raw/x-repos/wuyoscar__jev-skill.md
  - raw/x-repos/wuyoscar__jev-skill__evals-results.md
  - raw/x-repos/wuyoscar__jev-skill__docs-experiments-model-panel-readme.md
  - raw/x-repos/ThinkFlowLab__system1-agents.md
  - raw/x-repos/ThinkFlowLab__system1-agents__docs-benchmarks.md
jev_version: "jev-1.13.0"
summary: "Builder A/B tests of Jev added to agents: model routers, skill and tool pickers, approval gates, advice loops. Context and cost usually fell; accuracy and wall time often did not improve."
---

# Head-to-head: Jev inside agents, routers and tool gates

> **TL;DR** When builders A/B-tested a whole agent with and without Jev, the reliable gains were smaller context and lower cost. Task success usually stayed the same, sometimes dropped, and wall time often rose: an agent ignored an optional Jev tool (0/150), a cheap model alone matched a Jev router, a Claude subagent picked skills better, and gates over-denied without a written policy. Measure the whole agent, not the decision. Split from [[ideas/head-to-head]] on 2026-09-25.

Numbers as each builder published them, on `jev-1.13.0`, mostly small n and one run. **LOST** = the Jev arm lost on the stated goal. Patterns: [[ideas/patterns-agents]] P02 model and effort routing, P03 safety gates, P05 worker routing; [[ideas/patterns-interface]] P12 browser. Rows moved here from [[ideas/head-to-head]]: LiteLLM, JevRouter, langchain-skill-router.

## Routing a request to a model

| Task (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| Auto Router tier, 80 author-labelled prompts × 3 ([LiteLLM blog](https://docs.litellm.ai/blog/jev-auto-router-benchmark), Moe Khalil; **vendor**: LiteLLM ships the router) | Jev vs Claude Haiku 4.5 | Tier match 95.00% vs 73.75%; p50 126.81 vs 688.40 ms; classifier cost 96.1% lower ($0.0077 vs $0.1985 for 240 calls) | All | Labels and prompts by the same author, not reviewed; Jev 87.5% on long cases vs 97.5% short; final answers not graded |
| Model tier for 120 prompts, blind-scored by two judges ([tiershift](https://github.com/iamvatsalpatel/tiershift), iamvatsalpatel) | Jev router (11 questions in one call) vs always GPT-5.6 Sol, always DeepSeek Flash | Quality 4.77 at $7.95 per 1,000 prompts vs Sol 4.77 at $13.23 vs Flash **4.78 at $0.37**. Routing p50 180 ms, $0.0418 per 1,000; all 18 high-stakes prompts went to the mid tier | **LOST**: the cheap model alone matched, so routing added cost | Author's own reading; single-turn prompts. npm and PyPI ship 0.1.0, the benchmarked policy is repo 0.2.0 |
| Pi coding-agent router, 5 tasks, 2026-09-20 ([pi-jev-router](https://github.com/philippdubach/pi-jev-router), philippdubach) | One Jev request (5 questions) feeds a quality/cost/latency pick over the OpenRouter catalogue, vs fixed Sonnet 5 | All arms 5/5; cost per success $0.0399 vs $0.1144 | Cost | Jev's own cost excluded; the author says 5 tasks prove no general saving. OpenRouter base URL and model ID `verified`. npm `pi-jev-router` is another publisher's |
| Prompt router in a Claude Code plugin ([claude-jev](https://github.com/0x7067/claude-jev), 0x7067) | Jev predicts the task type from the prompt, against labels derived from transcripts | 34.6% on 1,613 prompts; humans agreed with the derived labels only 52.5% on 120; a coarse 3-way taxonomy scored 58.8%, below a constant guess (64.7%) | **LOST** | Same plugin's rule checker per edit: 22 of 247 accepted real edits blocked (17 from one repo's no-comments rule), 17/29 planted violations caught, 0/24 near-misses blocked, median 0.80 s. Compaction ~0.9-1.1 s vs ~117 s; the planted constraint survived all 56 sessions |

## Picking skills and tools

| Task (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| Skill routing in a bank-statement agent, 236 skills, 55 conversations × 5 turns ([langchain-skill-router](https://github.com/deyna256/langchain-skill-router), deyna256) | Jev picks the skills vs the full catalog in the agent's prompt | 25.8k vs 113.0k input tokens per turn (4.4× less); right skill in front of the model 85% vs 55%; correct answers 90% vs 88% (+1.5, 95% CI −1.5 to +4.7, p = 0.63) | Context size; accuracy tie | Generated data, one agent model. A wrong skill loaded: 72% correct vs 92% with the right one. An earlier run was 5 pts below the full catalog (six contradictory skills) |
| First-5 tool-call prediction, 10 Toolathlon tasks ([JevRouter](https://github.com/BillionsBobby/JevRouter), BillionsBobby) | vs DeepSeek V4.1 Flash | Position-wise hits 38% (serial) / 44% (decompose + thread) vs 24%; 1.58 s (serial) vs 8.65 s per task; $0.0058 vs ≈ $0.0407 per 10 tasks | All | Author-reported, n = 10; measures routing prediction, not task completion |
| 162 routing fixtures for roles, skills and tools ([agent-dispatcher](https://github.com/nahid-sparktales/agent-dispatcher), nahid-sparktales) | Jev as judge vs a Claude subagent vs keywords | Top-1 143/162 vs **158/162** vs 23/162; skill precision/recall 0.68/0.71 vs 0.74/0.90; tools a tie; Jev median 358 ms; five reruns 136-143 | **LOST** to the Claude subagent, so the author ships Jev off by default | The Claude column is a reconstruction; only 20 tool cases. The repo has since become a retrieval engine |
| 72 synthetic requests over 24 synthetic skills ([jev-agent-skill-router](https://github.com/GodsBoy/jev-agent-skill-router), GodsBoy) | Batched Choice rounds, then need / ambiguity / fit Nouls gate in code, vs a lexical baseline | 68/72 vs 51/72; 0 wrong routes, 18 sent to review, 1 invalid response; median 1,287 ms, p95 1,406 ms; 255,897 input tokens (≥ $0.0107) | Accuracy | Questions revised after an earlier run (64/72). The author reports the cookbook pin `jev-1.12` returned HTTP 400 on 2026-09-16: `unverified`; [[reference/models-and-pricing]] lists `jev-1.13.0` |
| Capability-doc tree search, 20 scenarios × 4 setups ([JCR](https://github.com/NiazMorshed2007/jcr), NiazMorshed2007) | Jev ranks each node's children (with a no-match option) vs the agent reading all docs | Agent input −85% with Opus 5, −23% with GPT-5.6 Sol; cost −67% and −16%. Sol slower with JCR in 19 of 20 scenarios (median 23.2 → 45.4 s) | Context, cost; **LOST** time | One run each, lookup only; only a chart is committed |
| Hermes skill suggestions, 292 skills ([typesafe-skill-router](https://github.com/DECRUX9812/typesafe-skill-router), DECRUX9812) | Port of the [[cookbooks/skill-suggestion]] cookbook (wide Choice with `none_of_these`, 3 gate Nouls) | ~0.6-1.2 s and ~$0.001 per turn; a second check (n = 12) p50 1.93 s, max 8.56 s; `fits` ~0.14 lower in Spanish; first 4 live turns: one miss, one off-target | Unclear | Its "budget for output tokens" `contradicts docs` (output is free, [[reference/models-and-pricing]]) |
| Tool steering in coding agents, 120 sessions: 6 models × 2 chess tasks × 5 runs ([jev-gateway](https://github.com/vinilana/jev-gateway), vinilana; companion jev-gateway-bench) | A Jev Choice over the agent's tools forces or hints a tool when confident, vs no routing | Bug fix: cheaper for every model (GPT-6 Astra and GPT-5.6 Sol −57% output tokens). Feature task: Opus 5 +61% input, +83% time; Sonnet 5 also worse; GPT-5.6 Luna solved 3/5 vs 5/5 without routing | Cost on bug fixes; **LOST** on the feature task | Adds a Jev call per turn (~0.5-1 s in our 2026-09-25 sweep note; not in the captured READMEs, which show one 712 ms key check: `unverified`); fails open. Author confirmed OpenCode Zen free and paid IDs live on 2026-09-24 |
| Judgment tools offered to an agent, 150 Pi tasks and a 3 × 3 outcome study ([jev-judge-mcp](https://github.com/PyModel/jev-judge-mcp), PyModel) | Jev as an optional or forced MCP tool | Optional: never called (0/150). Forced: +10.4 s median per task; Jev itself 465 ms median. Outcomes: Claude Code 6/9 in both arms, 4.6 s slower with Jev; Pi 6/8 in both, 85.8 s slower | **LOST** time; no accuracy gain | Paid, small samples; Jev's accuracy not measured |

## Approval and safety gates

| Task (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| 156 real mined shell commands, 2026-09-17 ([hermes-jev-approvals](https://github.com/anpicasso/hermes-jev-approvals), anpicasso) | Six questions in one request, combined by code rules, vs Hermes's small chat-LLM reviewer | 10 human prompts vs 42; 405 ms vs 3,968 ms average. **Without an operator policy** Jev denied 17 of 153, 8 of them routine Chrome kills, and denied or escalated plain `git push origin dev` until a policy clause was added (`--force` to main stays denied) | Fewer prompts, latency | Thresholds tuned on the same data; no production use |
| Same plugin, independent assay: 28 unique synthetic commands, 156 observations ([Bear Huddleston](https://bearhuddleston.dev/reports/jev-approvals-live-sandbox/)) | Jev plugin vs GPT-5.4 mini vs GPT-5.6 Luna reviewers | Mean reviewer time 1.062 / 1.319 / 1.198 s (Mini/Jev 1.24x, not the author's 9.8x). The plugin sent a 2,000-character prefix of a 2,909-character policy and approved a deletion the cut exception forbade; Mini and Luna denied it. Luna matched all 38 reviewed primary verdicts | Latency; **LOST** where input was cut | Already-seen cases, no holdout; cost is a published-rate estimate (Jev $0.00229 per 44 calls) |
| DeepSeek Harness tool gate ([dsh-jev](https://github.com/buberlo/dsh-jev), buberlo) | Jev narrows tools and checks each call, vs a rule written in the prompt | Author: no better than the prompt rule when the model complies (0/10 deletions), and slower. A weaker model tried deleting 10/10 times; Jev denied all 31 attempts. k3s replay: denied the broad allow-all patch (0.960) but also falsely denied a correct reset (0.940) | Tie; blocks a non-compliant model | One machine, one model, one day. npm 0.1.2 and 0.1.3 are broken |
| Coding-agent guard with a rules table first ([jev-kit](https://github.com/jonathanavis96/jev-kit), jonathanavis96) | One Jev question for the ambiguous rest; deny needs confidence ≥ 0.8 and margin ≥ 0.4 | General-risk rule 20/21; tier guard 56/57 (ECE 1.9% on 50 rows); search-intent guard 84.4% (ECE 12.6% on 32 rows). The 30-session A/B produced **zero denies**, so no savings shown. Browser clicks 9/9 at 314-486 ms median vs Sonnet 9/9 at 1.1-1.5 s vs Haiku 4/9 (n = 3 per cell). Wikipedia hops: 15/18 when named, **0/6** when not | Latency on clicks | Invasive install (hook on every tool, systemd timers). Author numbers, 2026-09-19 and 09-23 |

## Advice and decisions inside agent loops

| Task (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| Pre-declared pilot, 24 episodes, DeepSeek V4.1 Flash, advice at turns 3/6/9 ([jev-skill](https://github.com/wuyoscar/jev-skill), wuyoscar) | Agent with vs without Jev advice | Completed 12/12 vs **10/12** (0 Jev wins, 10 ties, 2 losses); cost $0.0051 → $0.0091; API time per episode 6.33 → 12.57 s. A 2026-09-22 rerun (4 pairs): 3/4 → 4/4 | **LOST** the pilot | Author: neither run proves a general benefit. Every request and response committed. Its BBH calibration: [[ideas/failure-reports-confidence]] |
| openJiuwen agents with a decision model in the model slot ([system1-agents](https://github.com/ThinkFlowLab/system1-agents), ThinkFlowLab) | Jev vs an unnamed chat model | Series of 2026-09-19: Blackjack 100 hands tie; ALFWorld 12 episodes 0.75 vs **0.917** at 1/16 the cost; 2048 (20 moves) 64 vs 80; flight booking 3/3 in every arm | Cost; **LOST** ALFWorld and 2048 | The README's "6× faster, 25× cheaper at the same score" rests on single episodes |

## How to read these

- **Measure the agent, not the call.** Jev's own latency (180-465 ms here) was never the problem; forced calls, extra turns and repeated navigation added seconds (jev-judge-mcp, JCR, wuyoscar). Browser agents show the same: [[ideas/head-to-head-benchmarks]] (Jevry).
- **Gains:** less context in front of the model (langchain-skill-router, JCR), cheaper bug fixes (jev-gateway), fewer human prompts (hermes-jev-approvals), cheaper routing classification (LiteLLM).
- **Losses:** a cheap model alone (tiershift), a Claude subagent (agent-dispatcher), a prompt rule (dsh-jev), no Jev at all (wuyoscar pilot, jev-gateway feature task). Effects depend on the model: routing helped GPT-5.6 Sol and hurt Opus 5.
- **Gates need a written policy** passed whole: without one Jev over-denied (hermes), and a truncated policy flipped a verdict (Bear Huddleston). Deterministic rules first ([[ideas/patterns-agents]] P03).
- Agents ignore optional tools: if Jev must run, call it from code, not as an offered tool (jev-judge-mcp).

## Related

- [[ideas/head-to-head]]: app-level comparisons and cascades; [[ideas/head-to-head-benchmarks]]: public sets; [[ideas/cost-ledger]]: cost per decision
- [[ideas/failure-reports]]: what broke; [[ideas/builds-agents]], [[ideas/tools-and-integrations]]: the tools themselves
- [[concepts/jev-with-coding-agents]]: what TypeSafe says about coding agents

## Sources

Links inline; raw captures listed in the frontmatter (private repo).