Agents: read the raw Markdown of this page, or start at llms.txt.
~/wiki/ideas
Builder lessons: what changed the result
TL;DR In builders' own runs (2026-09-16→24,
jev-1.13.0), the biggest gains came from what went intostate(deployment purpose, link targets, computed estimates), not from rewording. Split questions when answers combine by max or weights, not in all-must-pass screens. Cascade gates read on the test set are upper bounds. Each row is one builder's run, author-reported unless marked; re-measure on your own data. Split from Field reports: independent evaluations, critiques, open replicas on 2026-09-25.
Lessons
One builder's run each on jev-1.13.0 (2026-09-16→24), author-reported unless marked. Head-to-head numbers: Head-to-head: Jev against other models and methods; breakages: Failure reports: where Jev broke, lost, or was the wrong tool; cost and speed: Measurements, access routes and open replicas.
| Lesson | Who | Evidence | Check |
|---|---|---|---|
Put the deployment in state before rewording. One line saying what the assistant is for (a news publisher's reader assistant) took prompt-injection accuracy 89.7% → 96.5% and recall 74.9% → 95.1% at a plain 0.50 cut. AUC barely moved (0.9846 → 0.9927): the ranking was right, only the probabilities moved to where a fixed cut works. Without it, blatant overrides were caught (precision 99.0%), topic hijacks read as ordinary requests |
Gaurav Gosain, jev-sec-bench | 662 labelled messages, one request each, 2026-09-16; per-sample results published | verified in direction: literal reading (Jev 1.13 jaggedness: known failure modes §1), send what the judgment needs (State: what you send Jev). He also found it under-confident at the top (the 0.85 bucket was right 44/44) |
Evidence beats instructions. Same Choice and definitions; adding link destinations, Reply-To and attachment metadata to state took a ham/spam/phishing test 93.62% → 97.98%, phishing recall 85.71% → 98.43% (256 errors fixed, 6 introduced). A further wording change added little (98.64%) and caught fewer recent phishing (94.49% vs 95.31%). Detailed criteria helped on one corpus (95.96% → 98.33%), hurt on Ling-Spam (98.57% → 97.01%: announcements lacked subscription context) |
bitnovus, jev-spam-eval | 5,733-message test, mostly single runs; author: exploratory, specs shaped by earlier labelled errors, pretraining exposure unknown | verified in direction (State: what you send Jev). More instruction is not uniformly better: re-measure each wording change |
Word options as an operator would; send comparable estimates, not raw readings. Terse labels: give-way 0.46 vs maintain 0.41 for a conflicting emergency arrival; operational criteria ("climb or offset now so the emergency aircraft has a clear approach"): 0.70 / 0.05, same state. A raw faults.fuel: 0.83 got "keep cruising" 90 s from an empty tank; fuel minutes beside ETA fixed it. Runway exclusivity stays in code (0 double occupancies in stress runs) |
lbotinelly, Little Airways | single observations in a demo | verified: criteria carry the boundary cases (§1); numbers go in as computed comparisons (§2, numeric representations); Writing instructions and criteria that Jev reads correctly |
| Split for OR, keep conjunctions short. Two crisply worded interests gave 14 confident hits each on 299 arXiv papers; three vaguer ones never crossed 0.95. In review screening, where every criterion must hold, splitting compound criteria into more cut one topic's work saved 10.0% → 0.1%: each criterion is another veto | Eliot5566, Paper Radar | one day's papers; CLEF TAR replay | Qualifies @razeden0's split (Field reports: independent evaluations, critiques, open replicas): it helps when answers combine by max or weights, not when all must pass. Recall caveats: Failure reports: where Jev broke, lost, or was the wrong tool |
Move the judgments, keep the prose on an LLM. An Opus agent loop that chose sources and wrote reports cost $8–15/day for three or four topics (claude -p usage; the article says $13–15 over its last three days) and kept circling one theme (88 of 238 topics). With Jev screening each (source, question) pair and an LLM writing only prose: $1–2 per four-topic morning ($1.05 and $1.75 on the first two daily runs, Qwen prose included). Unanchored relevance questions passed anything sharing a term; an on-topic screen first cut one line from 20,573 questions (a 283 KB note) to ~900–1,900. Prose stayed hard: an independent Opus judge rated at most 1 of 3 notes publishable |
shimo4228, jev-research-pipeline, article | 11-run pilot (ended 2026-09-23), then daily; the article declines to compare costs until the Jev share is checked against a bill | Design verified (How to build software with System One: code owns the loop, generation elsewhere). Costs unverified: the loops differ in more than the model |
| Route speed is your concurrency. Four questions per Telegram post, 1,500 posts: ~85 s direct (2 pages at a time) vs ~35 s through OpenRouter's alpha Decisions API (24 requests in flight). On a channel with no paid ads it still marked two partner promotions as ads | gaborishka, Jev Wrapped | author's runs | Direct pacing followed the documented 1,200 rpm (verified, Models, aliases, pricing, rate limits, context); the gap is concurrency, not model speed (inferred). Ad vs own promotion is a boundary the criteria must spell out (§1). Numbers: Measurements, access routes and open replicas |
When keywords mark the answer, grep wins. siftr's filter kept 100% of alert lines at 46% of a labelled LogHub log; grep "fatal|error|fail" kept 100% at 23%. Its search put the fixing file in the top 5 for 82% of issues (BM25 52%, grep 22%) |
Bentlybro, siftr | frozen 202-issue SWE-bench Lite split; one log dataset; filter shipped as experimental |
verified in direction (When not to use Jev: rules, embeddings, trained classifiers, small and frontier LLMs: rules first). Use Jev where keywords cannot express the question. Tool: Builds: coding agents, harnesses and orchestration |
| Read vendor n=1 claims to the footnote. Yappy's page: a job form in 1m54s vs 13m14s and ~$0.24 vs ~$4.24 against a named competitor; Jev's five decisions 275–690 ms, $0.003. Its research page: Jev picked the right four fields but every write failed (Chrome's React inputs ignore values set through accessibility), confidence fell to 0.21 and the full agent (Claude Opus) finished; an earlier run that morning took 4m41s and $0.65 | Yappy, page, research | vendor, one run per column | The headline times the fallback agent, not Jev filling a form (the vendor says so). Sound hand-off rule: confidence under 0.25 or three no-op actions (Confidence-gated routing). Its "32k context" is the state-plus-longest-question budget; a request may carry 64k (Models, aliases, pricing, rate limits, context) |
| Cascade gates read on the test set are an upper bound. Banking77, 3,080 items: accept Jev at confidence ≥ 0.90, send the rest to Claude Opus 5 → 84.0% at $0.69 per 1,000 vs Opus alone 84.4% at $2.42; the author calls it an upper bound (gate read on the same items). Confidence ranked well but is not a probability of being right (≥ 0.99: 96.3% correct; < 0.5: 29.6%). Both models got 0/40 on an intent whose label name hides its meaning | Kenny Rogers, OpenRouter blog | vendor (sells both models); one afternoon | verified in direction: confidence summarises the distribution's shape; gate by stakes on your labels (Confidence vs probability, Confidence-gated routing) |
| Same lesson, second source: a 0.80 Jev gate with GPT-5.6 Terra as fallback matched Terra at 26–28% of its cost; cutoffs untuned but scored on the same items, "so treat the exact figures as an estimate" | Adel Dahani, AY Automate | agency blog with sales pitches; 791 decisions, harness and results published; no client use of Jev | Pick the gate on a validation split, confirm on held-out data (Testing and evaluating a Jev workflow) |
What this means when advising
- Fix
statebefore the wording: add the deployment's purpose, the evidence a person would look at, and numbers already turned into comparisons. - Re-measure every wording or criteria change on labelled data: the same edit helped one corpus and hurt another (Spam Eval).
- Keep generation, invariants and arithmetic in code or an LLM; let Jev pick (How to build software with System One).
- Treat vendor and reseller figures as marketing until the footnotes check out; prefer published harnesses and per-sample results.
Related
- Field reports: independent evaluations, critiques, open replicas — what held up and what broke; Measurements, access routes and open replicas — cost and speed from these runs; Head-to-head: Jev against other models and methods — the same studies scored against other models; Failure reports: where Jev broke, lost, or was the wrong tool — where they broke
- Writing instructions and criteria that Jev reads correctly, State: what you send Jev, Confidence-gated routing — the official guidance each lesson checks against
Sources
Repos, articles and vendor pages in this page's frontmatter; URLs inline. Captured 2026-09-25 (awesome-jev review, owner-approved 2026-09-25).