---
title: "Builder lessons: what changed the result"
type: community
source_tier: community
tags: [field-reports, lessons, state, question-design, cascades]
created: 2026-09-25
updated: 2026-09-25
confidence: medium
sources:
  - raw/x-repos/Gaurav-Gosain__jev-sec-bench.md
  - raw/x-repos/bitnovus__jev-spam-eval.md
  - raw/x-repos/lbotinelly__jev-little-airways.md
  - raw/x-repos/Eliot5566__JEV-Paper-Radar.md
  - raw/x-repos/shimo4228__jev-research-pipeline.md
  - raw/community/dev-to-shimo4228-moving-my-research-pipelines-judgment-calls-from-an-llm-to-jev-4ncj.md
  - raw/x-repos/gaborishka__jev-wrapped.md
  - raw/x-repos/Bentlybro__siftr.md
  - raw/x-repos/Bentlybro__siftr__benchmarks.md
  - raw/community/yappy-biz-jev.md
  - raw/community/yappy-biz-research-yappy-vs-heyclicky.md
  - raw/community/openrouter-ai-blog-insights-jev-vs-claude-opus-5-classification.md
  - raw/community/ayautomate-com-blog-jev-vs-llm-benchmark.md
jev_version: "jev-1.13.0"
summary: "Builders' own before/after runs: context in state beat rewording, estimates beat raw numbers, conjunctions veto, test-set cascade gates are upper bounds, grep wins on keyword logs."
---

# Builder lessons: what changed the result

> **TL;DR** In builders' own runs (2026-09-16→24, `jev-1.13.0`), the biggest gains came from **what went into `state`** (deployment purpose, link targets, computed estimates), not from rewording. Split questions when answers combine by max or weights, not in all-must-pass screens. Cascade gates read on the test set are upper bounds. Each row is one builder's run, author-reported unless marked; re-measure on your own data. Split from [[ideas/field-reports]] on 2026-09-25.

## Lessons

One builder's run each on `jev-1.13.0` (2026-09-16→24), author-reported unless marked. Head-to-head numbers: [[ideas/head-to-head]]; breakages: [[ideas/failure-reports]]; cost and speed: [[ideas/measurements]].

| Lesson | Who | Evidence | Check |
|---|---|---|---|
| **Put the deployment in `state` before rewording.** One line saying what the assistant is for (a news publisher's reader assistant) took prompt-injection accuracy **89.7% → 96.5%** and recall **74.9% → 95.1%** at a plain 0.50 cut. AUC barely moved (0.9846 → 0.9927): the ranking was right, only the probabilities moved to where a fixed cut works. Without it, blatant overrides were caught (precision 99.0%), topic hijacks read as ordinary requests | Gaurav Gosain, [jev-sec-bench](https://github.com/Gaurav-Gosain/jev-sec-bench) | 662 labelled messages, one request each, 2026-09-16; per-sample results published | `verified` in direction: literal reading ([[concepts/jaggedness-jev-1-13]] §1), send what the judgment needs ([[concepts/state]]). He also found it under-confident at the top (the 0.85 bucket was right 44/44) |
| **Evidence beats instructions.** Same Choice and definitions; adding link destinations, Reply-To and attachment metadata to `state` took a ham/spam/phishing test **93.62% → 97.98%**, phishing recall **85.71% → 98.43%** (256 errors fixed, 6 introduced). A further wording change added little (98.64%) and caught fewer recent phishing (94.49% vs 95.31%). Detailed criteria helped on one corpus (95.96% → 98.33%), hurt on Ling-Spam (98.57% → 97.01%: announcements lacked subscription context) | bitnovus, [jev-spam-eval](https://github.com/bitnovus/jev-spam-eval) | 5,733-message test, mostly single runs; author: exploratory, specs shaped by earlier labelled errors, pretraining exposure unknown | `verified` in direction ([[concepts/state]]). More instruction is not uniformly better: re-measure each wording change |
| **Word options as an operator would; send comparable estimates, not raw readings.** Terse labels: give-way **0.46** vs maintain 0.41 for a conflicting emergency arrival; operational criteria ("climb or offset now so the emergency aircraft has a clear approach"): **0.70 / 0.05**, same state. A raw `faults.fuel: 0.83` got "keep cruising" 90 s from an empty tank; fuel minutes beside ETA fixed it. Runway exclusivity stays in code (0 double occupancies in stress runs) | lbotinelly, [Little Airways](https://github.com/lbotinelly/jev-little-airways) | single observations in a demo | `verified`: criteria carry the boundary cases (§1); numbers go in as computed comparisons (§2, numeric representations); [[guides/writing-instructions-and-criteria]] |
| **Split for OR, keep conjunctions short.** Two crisply worded interests gave 14 confident hits each on 299 arXiv papers; three vaguer ones never crossed 0.95. In review screening, where every criterion must hold, splitting compound criteria into more cut one topic's work saved **10.0% → 0.1%**: each criterion is another veto | Eliot5566, [Paper Radar](https://github.com/Eliot5566/JEV-Paper-Radar) | one day's papers; CLEF TAR replay | Qualifies @razeden0's split ([[ideas/field-reports]]): it helps when answers combine by max or weights, not when all must pass. Recall caveats: [[ideas/failure-reports]] |
| **Move the judgments, keep the prose on an LLM.** An Opus agent loop that chose sources and wrote reports cost $8–15/day for three or four topics (`claude -p` usage; the article says $13–15 over its last three days) and kept circling one theme (88 of 238 topics). With Jev screening each (source, question) pair and an LLM writing only prose: **$1–2 per four-topic morning** ($1.05 and $1.75 on the first two daily runs, Qwen prose included). Unanchored relevance questions passed anything sharing a term; an on-topic screen first cut one line from 20,573 questions (a 283 KB note) to ~900–1,900. Prose stayed hard: an independent Opus judge rated at most 1 of 3 notes publishable | shimo4228, [jev-research-pipeline](https://github.com/shimo4228/jev-research-pipeline), [article](https://dev.to/shimo4228/moving-my-research-pipelines-judgment-calls-from-an-llm-to-jev-a-judgment-only-model-4ncj) | 11-run pilot (ended 2026-09-23), then daily; the article declines to compare costs until the Jev share is checked against a bill | Design `verified` ([[concepts/how-to-build]]: code owns the loop, generation elsewhere). Costs `unverified`: the loops differ in more than the model |
| **Route speed is your concurrency.** Four questions per Telegram post, 1,500 posts: ~85 s direct (2 pages at a time) vs ~35 s through OpenRouter's alpha Decisions API (24 requests in flight). On a channel with no paid ads it still marked two partner promotions as ads | gaborishka, [Jev Wrapped](https://github.com/gaborishka/jev-wrapped) | author's runs | Direct pacing followed the documented 1,200 rpm (`verified`, [[reference/models-and-pricing]]); the gap is concurrency, not model speed (inferred). Ad vs own promotion is a boundary the criteria must spell out (§1). Numbers: [[ideas/measurements]] |
| **When keywords mark the answer, grep wins.** siftr's `filter` kept 100% of alert lines at 46% of a labelled LogHub log; `grep "fatal\|error\|fail"` kept 100% at 23%. Its `search` put the fixing file in the top 5 for **82%** of issues (BM25 52%, grep 22%) | Bentlybro, [siftr](https://github.com/Bentlybro/siftr) | frozen 202-issue SWE-bench Lite split; one log dataset; `filter` shipped as experimental | `verified` in direction ([[syntheses/when-not-to-use-jev]]: rules first). Use Jev where keywords cannot express the question. Tool: [[ideas/builds-agents]] |
| **Read vendor n=1 claims to the footnote.** Yappy's page: a job form in 1m54s vs 13m14s and ~$0.24 vs ~$4.24 against a named competitor; Jev's five decisions 275–690 ms, $0.003. Its research page: Jev picked the right four fields but every write failed (Chrome's React inputs ignore values set through accessibility), confidence fell to 0.21 and the full agent (Claude Opus) finished; an earlier run that morning took 4m41s and $0.65 | Yappy, [page](https://yappy.biz/jev/), [research](https://yappy.biz/research/yappy-vs-heyclicky/) | **vendor**, one run per column | The headline times the fallback agent, not Jev filling a form (the vendor says so). Sound hand-off rule: confidence under 0.25 or three no-op actions ([[patterns/confidence-routing]]). Its "32k context" is the `state`-plus-longest-question budget; a request may carry 64k ([[reference/models-and-pricing]]) |
| **Cascade gates read on the test set are an upper bound.** Banking77, 3,080 items: accept Jev at confidence ≥ 0.90, send the rest to Claude Opus 5 → **84.0%** at $0.69 per 1,000 vs Opus alone 84.4% at $2.42; the author calls it an upper bound (gate read on the same items). Confidence ranked well but is not a probability of being right (≥ 0.99: 96.3% correct; < 0.5: 29.6%). Both models got 0/40 on an intent whose label name hides its meaning | Kenny Rogers, [OpenRouter blog](https://openrouter.ai/blog/insights/jev-vs-claude-opus-5-classification/) | **vendor** (sells both models); one afternoon | `verified` in direction: `confidence` summarises the distribution's shape; gate by stakes on your labels ([[concepts/confidence]], [[patterns/confidence-routing]]) |
| Same lesson, second source: a 0.80 Jev gate with GPT-5.6 Terra as fallback matched Terra at 26–28% of its cost; cutoffs untuned but scored on the same items, "so treat the exact figures as an estimate" | Adel Dahani, [AY Automate](https://www.ayautomate.com/blog/jev-vs-llm-benchmark) | agency blog with sales pitches; 791 decisions, harness and results published; no client use of Jev | Pick the gate on a validation split, confirm on held-out data ([[guides/testing-and-evaluation]]) |

## What this means when advising

- Fix `state` before the wording: add the deployment's purpose, the evidence a person would look at, and numbers already turned into comparisons.
- Re-measure every wording or criteria change on labelled data: the same edit helped one corpus and hurt another (Spam Eval).
- Keep generation, invariants and arithmetic in code or an LLM; let Jev pick ([[concepts/how-to-build]]).
- Treat vendor and reseller figures as marketing until the footnotes check out; prefer published harnesses and per-sample results.

## Related

- [[ideas/field-reports]] — what held up and what broke; [[ideas/measurements]] — cost and speed from these runs; [[ideas/head-to-head]] — the same studies scored against other models; [[ideas/failure-reports]] — where they broke
- [[guides/writing-instructions-and-criteria]], [[concepts/state]], [[patterns/confidence-routing]] — the official guidance each lesson checks against

## Sources

Repos, articles and vendor pages in this page's frontmatter; URLs inline. Captured 2026-09-25 (awesome-jev review, owner-approved 2026-09-25).