---
title: "Failure reports: where Jev broke, lost, or was the wrong tool"
type: community
source_tier: community
tags: [failure-reports, limitations, jaggedness, advising]
created: 2026-09-23
updated: 2026-09-23
confidence: medium
sources:
  - raw/x-repos/erik-dunteman__ChatJev.md
  - raw/x/yurinakanishi33-2101946125385437622.md
  - raw/x-repos/RINNECODER__jev-behavior-study.md
  - raw/x-repos/denikuchero__jev-chess-lab.md
  - raw/x/GodName794-2102199799160475707.md
  - raw/x/davidmadurzak-2102048463634526637.md
  - raw/x/umezawakanta13-2102062482991001689.md
  - raw/x-repos/sosopop__jev_stock.md
  - raw/community/primeline-cc-blog-typesafe-jev-pre-registered-test.md
  - raw/x/zhengli-2101889473651065086.md
  - raw/x/vwapster-2102012812906094722.md
  - raw/x/hw40-2102332459203653977.md
  - raw/x/ranjankumar-2101953564834934999.md
  - raw/x/zeroxmin-2101940446637531634.md
  - raw/x-repos/duckegg0623-create__jev-wechat-live.md
  - raw/x/toowitter-2102197694622319014.md
  - raw/x/satto_sann-2102009241414619139.md
  - raw/x/voxmenthe-2102289186560589852.md
  - raw/x-repos/AbdelStark__jev-benchmarks.md
  - raw/community/reddit-com-r-themachinelearning-comments-1wn2gn8-jev-demos-are-misleading.md
jev_version: "jev-1.13.0"
summary: "Community reports (2026-09-16→22) where Jev failed, lost to a baseline or was the wrong tool, grouped by failure kind, with the documented weakness each matches."
---

# Failure reports: where Jev broke, lost, or was the wrong tool

> **TL;DR** Most reported failures are Jev asked to plan, count, generate, move in real time, or answer something the `state` lacks. When code builds the candidates and Jev picks, the same builders report wins. All n≈1. Earlier failures: [[ideas/field-reports]].

Marks check against [[concepts/jaggedness-jev-1-13]] (§n) and reference/. Numbers are the builder's own.

## Generation or sampling misuse

| Report | What broke | Matches | Takeaway |
|---|---|---|---|
| [ChatJev](https://github.com/erik-dunteman/ChatJev), erik-dunteman: Choice over next tokens in a loop | Author: it "kinda works". No numbers | §9 generation, `verified` | Generation needs a generative model. [[ideas/patterns-agents]] P01 |
| Haiku by Choice, [@yurinakanishi33](https://x.com/yurinakanishi33/status/2101946125385437622) | Her correction: code pre-built the 5/7-mora lines and re-sampled Choice's top 3. Unconstrained (73 kana, 17 draws): nonsense | §9, `verified` | Check if a demo's code did the generating (P01) |

## Arithmetic, numbers, spatial state and time series

| Report | What broke | Matches | Takeaway |
|---|---|---|---|
| [jev-behavior-study](https://github.com/RINNECODER/jev-behavior-study), RINNECODER: 11,621 requests, synthetic | Arithmetic: correct option first 95/108, fourth 62/108. Letter counts 117/216. Conflicting records mid-text at 1,024 records: 0/6. Missing-link "unknown": 0/6 at every chain length. Car wash: "walk" in all 1,000 calls though both sub-checks were right | §2, §4, §5 `verified`. Option-order effect `unverified` (not in docs) | Count in code; shuffle options in evals; test "unknown" cases; right sub-checks ≠ right action |
| Same study: 3D city and Snake | City: 0/12 tasks, "straight" on all 522 steering calls. Snake 8-food: direct 0/32, code-guarded 3/32, pathfinding 27/32. With code-computed route facts: 16/16 | §2, §4 `verified` in direction | Code computes the geometry, Jev picks. [[ideas/patterns-data]] P24 |
| [Jev Chess Lab](https://github.com/denikuchero/jev-chess-lab), denikuchero, via OpenRouter | Jev alone lost 2 games and stalled 1 (hit the call limit), giving away pieces. A Jev self-review loop changed its first choice on 16/23 turns and still lost (115 calls, $0.008995). Won only when picking from a Stockfish shortlist | Price `verified`: 214,165 input tok × $0.042/Mtok ≈ $0.0090 ([[reference/models-and-pricing]]) | Self-critique adds cost, not lookahead; let an engine build the shortlist (P24) |
| Gomoku, [@GodName794](https://x.com/GodName794/status/2102199799160475707) | Jev alone lost; with a code attack/defence filter on candidates the system beat him, credited to the whole system | §4 `verified` | Test with and without Jev; high probability ≠ replies checked |
| [jev_stock](https://github.com/sosopop/jev_stock), sosopop: HK stocks up/flat/down for T+1 | 54/120 = 45.0% over 4 stocks (per stock 30.0–56.7%). Author: `down: 0.68` is not a 68% win rate. Retrospective calls may leak history the model already knows | Time series `verified` poor ([[ideas/patterns-data]] P25) | Not investment advice. Past-only `state`, forward tests, simple baselines |

## Speed: too slow for games, or fast but less accurate

| Report | What broke | Matches | Takeaway |
|---|---|---|---|
| [Ms. Pac-Man](https://x.com/davidmadurzak/status/2102048463634526637), David Madurzak | Needs ~70 ms per move; he measured ~650 ms and paused the game per call. Poor maze navigation from raw state until he added move hints (§4, §5) | Latency `unverified`: docs publish none ([[concepts/workflow-evals]]) | No hard real-time loops. Tick-paused or turn-based games only |
| [Mario 1-1](https://x.com/umezawakanta13/status/2102062482991001689), @umezawakanta13 | Died at the first enemy (before a dodge assist). Browser round trip median 976.8 ms over 5 responses (not pure inference) | `unverified` | Budget the browser round trip, not only model time |
| [primeline.cc](https://primeline.cc/blog/typesafe-jev-pre-registered-test) (Robin), pre-registered test vs Claude Haiku 4.5 | Knowledge notes: Haiku won at every threshold (98.9% vs 98.6% on the 356 items Jev kept). Jev fell from 97.0% (shortest quarter) to 81.1% (longest); 62% of errors were one preference/fact mix-up; one extra narrow Noul fixed 23/53. Jev won commit-type, 65.7% vs 54.8% | §1 `verified` for the one-judgment-per-question fix ([[patterns/composite-scoring]]) | Where accuracy matters an LLM can win. Read wrong answers, add a narrow question. [[ideas/head-to-head]] |

## Cost surprises

| Report | What happened | Matches | Takeaway |
|---|---|---|---|
| [@zhengli](https://x.com/zhengli/status/2101889473651065086) | One real task used up a $5/month budget. Lesson: read the official skill, use `state` well, group questions | `verified`: `state` is ingested once per request, so group questions ([[patterns/fan-out]], [[reference/agent-skill]]) | Count calls × `state` size before quoting |
| [@vwapster](https://x.com/vwapster/status/2102012812906094722), one trading bot | Cheap per call, "going to add up fast". Numbers in an uncaptured image | `unverified` | Repeat decisions: price per hour (see [[ideas/field-reports]]). P25 poor |
| Two-step pipeline (RINNECODER) | Check-then-choose: 288/288, same as direct, at ~2.2x input tokens | — | Extra calls need a measured gain |
| Counterpoint: [@hw40](https://x.com/hw40/status/2102332459203653977) | 36,637 requests over two days for $2.49 | `unverified`: 62.6M tok × $0.042 = $2.63, so some was probably free output (inferred) | Grouped questions over a filtered `state` stay cheap |

## Confidence misread

| Report | Claim | Matches | Takeaway |
|---|---|---|---|
| [Ranjan Kumar](https://x.com/ranjankumar/status/2101953564834934999) | Trust ordering, not magnitude; a fixed 0.85 gate ignores your prevalence and cost ratio; fit a calibration map on a few hundred own labels | Advice `verified` in direction ([[concepts/confidence]]). "Numbers not trustworthy" `contradicts docs`. No per-customer adaptation `unverified` | Routing/ranking: use argmax. Gating: calibrate on the project's own data first. [[ideas/patterns-agents]] P06 |
| primeline.cc | ≥0.9 confidence: ~92% right on 73% of items; below 0.8 about 50%. Calibration error by type: Noul 0.012, Choice 0.086, Score 0.254 (slightly overconfident) | Noul has no `confidence` field: `verified` ([[reference/openapi-schemas]]). CJK caveat credited to the jaggedness doc: `contradicts docs` (not there) | Prefer Noul for gates. Never reuse a threshold across question types (§8) |
| Same article: yelban replaying [fast-jev-compaction](https://github.com/tamaratran/fast-jev-compaction) | None of 256 "keep" scores reached 0.3. A stub that always answers zero gave the same outcome (87.7% vs 88.5% reduction). The plugin sent only the result's length and tool name, never the text | §5 in spirit `verified`: the content was missing | Flat low scores usually mean `state` lacks the evidence; compare against a stub. P07 |
| [jev-benchmarks](https://github.com/AbdelStark/jev-benchmarks) (AbdelStark), pre-registered pilot vs local GLiNER2.5, 100 items per set | Jev ahead on AG News, Banking77 (0.910, 0.870 vs 0.700, 0.610). 6-label emotion: accuracy tie (0.480 vs 0.440), Brier 0.846 vs 0.668, true label at probability 0 for 16% | Docs call probabilities calibrated ([[concepts/confidence]]): per-task exception, n=100, `unverified` | Check calibration per label set before gating; fuzzy labels can be confidently wrong |

## Wrong role, or options that don't fit

| Report | What broke | Matches | Takeaway |
|---|---|---|---|
| [@hrishioa](https://x.com/hrishioa/status/2101842370052669903) (Southbridge), quoted by [@zeroxmin](https://x.com/zeroxmin/status/2101940446637531634) | Across a few thousand hours of agent runs: the best tool he tested for progress and time-to-finish, "dangerous" for detecting harmful commands, weak at spotting lazy models. Article not captured | `unverified`. Fits §6 adversarial content | Observer, not the only security gate; keep deterministic allow-lists (P03, P15) |
| [jev-wechat-live](https://github.com/duckegg0623-create/jev-wechat-live) (abandoned), via OpenRouter | Reading subtext in WeChat messages: "reply with a clarifying question" in 6/11, "irritated" in 5/11, single emoji unreadable. Added context barely helped. Author: options were a linguist's taxonomy; with none fitting, Jev picked the nearest safe one | §1, §7 `verified` in direction | Options must cover reality, plus "other/none". [[ideas/patterns-business]] P20 |
| [@hw40](https://x.com/hw40/status/2102332459203653977), SEO tools | Under-called commercial pages as informational (25% of 263 vs an outside classifier). A citation rubric mixed two questions (33% → 64–70% after splitting). On a single-topic site, generic anchors scored high until a "specific" check was added (0.82 → 0.94) | §1 and "several judgments in one question" `verified` | Split rubrics; check against outside ground truth; log biases, don't threshold them away. [[ideas/patterns-business]] P32 |
| primeline.cc | A request phrased as a passing remark was missed 42.5% of the time until the question said such remarks count (0%). Two-stage narrowing was 12.9 points worse when broad groups overlapped | §1 `verified`. Narrowing penalty `unverified` | Name edge cases in the instruction. Don't assume hierarchy helps |
| [Reddit thread](https://www.reddit.com/r/TheMachineLearning/comments/1wn2gn8/jev_demos_are_misleading_says_developer/), 2026-09-23: critique, not evidence | Demos framed Jev as an agent (cars, games), not a classifier. A commenter: one forward pass caps what it can classify. Another: would old pattern matching do? | Jev does not choose its own next action ([[concepts/system-one]]) `verified`. Single-pass ceiling `unverified` | Pitch it as a router in a code-owned loop; baseline rules first ([[syntheses/when-not-to-use-jev]]) |

## A simpler tool won

| Report | What happened | Takeaway |
|---|---|---|
| [@toowitter](https://x.com/toowitter/status/2102197694622319014), unawair v3 | Explaining the device logic to Jev simplified the rules until a rule engine could do it; outdoor-temperature response much better | Writing the prompt may show plain code suffices: still a win. P24 |
| [@gigabit_million](https://x.com/gigabit_million/status/2101813176887390398), quoted by [@satto_sann](https://x.com/satto_sann/status/2102009241414619139) | Rebuilt a Jev classifier in BERT. It worked, but building and updating the criteria is expert work, and satto_sann warns "quick classifier" apps may be a build trap | Fixed labels plus a need for transparent criteria → BERT. Else Jev's value is time saved. P34 (inferred) |
| [@voxmenthe](https://x.com/voxmenthe/status/2102289186560589852), code-search-jev | Naive BM25 + Jev wasn't good enough. He added four lexical channels to build candidates, then Jev routes and scores regions: ~2.5 requests, ~18k input tok, ~750 ms warm | His ~$0.0015/query `contradicts docs` arithmetic (18k × $0.042/Mtok ≈ $0.00076; gateway or estimate?). Candidate recall comes first. [[ideas/patterns-data]] P17 |

## Integration and access

- Chess and WeChat ran via OpenRouter (its terms apply, [[ideas/measurements]]); API errors cut games short: retry, checkpoint ([[reference/rate-limits-and-errors]]).

## Checklist before recommending Jev

1. Does the answer need generation, counting, date maths, geometry or lookahead? Put that in code and let Jev pick from the candidates (§2–§4, §9).
2. Is the needed evidence actually in `state`? Test against a stub that always answers the same.
3. Do the options cover reality, with "other/none", one judgment each? Shuffle the order in evals.
4. Faster than reported round trips (260 ms–1 s)? Pause per decision or skip Jev.
5. Price it: calls/hour × `state` tokens × $0.042/Mtok, with grouped questions.
6. Gate on confidence only after calibrating on the project's own labels; Noul for gates.
7. Baseline rules, BERT, a solver or a small LLM on the same items; use Jev where it wins or ties cheaper.
8. Never the sole security gate; no trading recommendations.

## Related

- [[ideas/field-reports]], [[ideas/head-to-head]], [[ideas/measurements]] — earlier failures, comparisons, numbers
- [[concepts/jaggedness-jev-1-13]] — official failure modes; [[concepts/confidence]]; [[guides/testing-and-evaluation]]
- [[ideas/consult]] — the advising procedure; [[ideas/patterns]] — fit verdicts

## Sources

Files in frontmatter `sources:`, captured by 2026-09-23; original URLs inline.