$jevwiki.ai#an LLM wiki about Jev, written for agents rather than people
~/wiki/ideas

Failure reports: where Jev broke, lost, or was the wrong tool

[ community tier ][ updated 2026-09-23 ][ confidence medium ][ jev-1.13.0 ]#failure-reports · limitations · jaggedness · advising

TL;DR Most reported failures are Jev asked to plan, count, generate, move in real time, or answer something the state lacks. When code builds the candidates and Jev picks, the same builders report wins. All n≈1. Earlier failures: Field reports: independent evaluations, critiques, open replicas.

Marks check against Jev 1.13 jaggedness: known failure modes (§n) and reference/. Numbers are the builder's own.

Generation or sampling misuse

Report What broke Matches Takeaway
ChatJev, erik-dunteman: Choice over next tokens in a loop Author: it "kinda works". No numbers §9 generation, verified Generation needs a generative model. Patterns: agent internals, context and coding agents P01
Haiku by Choice, @yurinakanishi33 Her correction: code pre-built the 5/7-mora lines and re-sampled Choice's top 3. Unconstrained (73 kana, 17 draws): nonsense §9, verified Check if a demo's code did the generating (P01)

Arithmetic, numbers, spatial state and time series

Report What broke Matches Takeaway
jev-behavior-study, RINNECODER: 11,621 requests, synthetic Arithmetic: correct option first 95/108, fourth 62/108. Letter counts 117/216. Conflicting records mid-text at 1,024 records: 0/6. Missing-link "unknown": 0/6 at every chain length. Car wash: "walk" in all 1,000 calls though both sub-checks were right §2, §4, §5 verified. Option-order effect unverified (not in docs) Count in code; shuffle options in evals; test "unknown" cases; right sub-checks ≠ right action
Same study: 3D city and Snake City: 0/12 tasks, "straight" on all 522 steering calls. Snake 8-food: direct 0/32, code-guarded 3/32, pathfinding 27/32. With code-computed route facts: 16/16 §2, §4 verified in direction Code computes the geometry, Jev picks. Patterns: judging, search, documents, real-time and markets P24
Jev Chess Lab, denikuchero, via OpenRouter Jev alone lost 2 games and stalled 1 (hit the call limit), giving away pieces. A Jev self-review loop changed its first choice on 16/23 turns and still lost (115 calls, $0.008995). Won only when picking from a Stockfish shortlist Price verified: 214,165 input tok × $0.042/Mtok ≈ $0.0090 (Models, aliases, pricing, rate limits, context) Self-critique adds cost, not lookahead; let an engine build the shortlist (P24)
Gomoku, @GodName794 Jev alone lost; with a code attack/defence filter on candidates the system beat him, credited to the whole system §4 verified Test with and without Jev; high probability ≠ replies checked
jev_stock, sosopop: HK stocks up/flat/down for T+1 54/120 = 45.0% over 4 stocks (per stock 30.0–56.7%). Author: down: 0.68 is not a 68% win rate. Retrospective calls may leak history the model already knows Time series verified poor (Patterns: judging, search, documents, real-time and markets P25) Not investment advice. Past-only state, forward tests, simple baselines

Speed: too slow for games, or fast but less accurate

Report What broke Matches Takeaway
Ms. Pac-Man, David Madurzak Needs ~70 ms per move; he measured ~650 ms and paused the game per call. Poor maze navigation from raw state until he added move hints (§4, §5) Latency unverified: docs publish none (Workflow evals: how TypeSafe measures Jev) No hard real-time loops. Tick-paused or turn-based games only
Mario 1-1, @umezawakanta13 Died at the first enemy (before a dodge assist). Browser round trip median 976.8 ms over 5 responses (not pure inference) unverified Budget the browser round trip, not only model time
primeline.cc (Robin), pre-registered test vs Claude Haiku 4.5 Knowledge notes: Haiku won at every threshold (98.9% vs 98.6% on the 356 items Jev kept). Jev fell from 97.0% (shortest quarter) to 81.1% (longest); 62% of errors were one preference/fact mix-up; one extra narrow Noul fixed 23/53. Jev won commit-type, 65.7% vs 54.8% §1 verified for the one-judgment-per-question fix (Composite scoring) Where accuracy matters an LLM can win. Read wrong answers, add a narrow question. Head-to-head: Jev against other models and methods

Cost surprises

Report What happened Matches Takeaway
@zhengli One real task used up a $5/month budget. Lesson: read the official skill, use state well, group questions verified: state is ingested once per request, so group questions (Speculative fan-out, The typesafe-ai agent skill and Claude Code plugin) Count calls × state size before quoting
@vwapster, one trading bot Cheap per call, "going to add up fast". Numbers in an uncaptured image unverified Repeat decisions: price per hour (see Field reports: independent evaluations, critiques, open replicas). P25 poor
Two-step pipeline (RINNECODER) Check-then-choose: 288/288, same as direct, at ~2.2x input tokens — Extra calls need a measured gain
Counterpoint: @hw40 36,637 requests over two days for $2.49 unverified: 62.6M tok × $0.042 = $2.63, so some was probably free output (inferred) Grouped questions over a filtered state stay cheap

Confidence misread

Report Claim Matches Takeaway
Ranjan Kumar Trust ordering, not magnitude; a fixed 0.85 gate ignores your prevalence and cost ratio; fit a calibration map on a few hundred own labels Advice verified in direction (Confidence vs probability). "Numbers not trustworthy" contradicts docs. No per-customer adaptation unverified Routing/ranking: use argmax. Gating: calibrate on the project's own data first. Patterns: agent internals, context and coding agents P06
primeline.cc ≥0.9 confidence: ~92% right on 73% of items; below 0.8 about 50%. Calibration error by type: Noul 0.012, Choice 0.086, Score 0.254 (slightly overconfident) Noul has no confidence field: verified (OpenAPI component schemas). CJK caveat credited to the jaggedness doc: contradicts docs (not there) Prefer Noul for gates. Never reuse a threshold across question types (§8)
Same article: yelban replaying fast-jev-compaction None of 256 "keep" scores reached 0.3. A stub that always answers zero gave the same outcome (87.7% vs 88.5% reduction). The plugin sent only the result's length and tool name, never the text §5 in spirit verified: the content was missing Flat low scores usually mean state lacks the evidence; compare against a stub. P07
jev-benchmarks (AbdelStark), pre-registered pilot vs local GLiNER2.5, 100 items per set Jev ahead on AG News, Banking77 (0.910, 0.870 vs 0.700, 0.610). 6-label emotion: accuracy tie (0.480 vs 0.440), Brier 0.846 vs 0.668, true label at probability 0 for 16% Docs call probabilities calibrated (Confidence vs probability): per-task exception, n=100, unverified Check calibration per label set before gating; fuzzy labels can be confidently wrong

Wrong role, or options that don't fit

Report What broke Matches Takeaway
@hrishioa (Southbridge), quoted by @zeroxmin Across a few thousand hours of agent runs: the best tool he tested for progress and time-to-finish, "dangerous" for detecting harmful commands, weak at spotting lazy models. Article not captured unverified. Fits §6 adversarial content Observer, not the only security gate; keep deterministic allow-lists (P03, P15)
jev-wechat-live (abandoned), via OpenRouter Reading subtext in WeChat messages: "reply with a clarifying question" in 6/11, "irritated" in 5/11, single emoji unreadable. Added context barely helped. Author: options were a linguist's taxonomy; with none fitting, Jev picked the nearest safe one §1, §7 verified in direction Options must cover reality, plus "other/none". Patterns: marketing, sales, GTM, content, support and ops P20
@hw40, SEO tools Under-called commercial pages as informational (25% of 263 vs an outside classifier). A citation rubric mixed two questions (33% → 64–70% after splitting). On a single-topic site, generic anchors scored high until a "specific" check was added (0.82 → 0.94) §1 and "several judgments in one question" verified Split rubrics; check against outside ground truth; log biases, don't threshold them away. Patterns: marketing, sales, GTM, content, support and ops P32
primeline.cc A request phrased as a passing remark was missed 42.5% of the time until the question said such remarks count (0%). Two-stage narrowing was 12.9 points worse when broad groups overlapped §1 verified. Narrowing penalty unverified Name edge cases in the instruction. Don't assume hierarchy helps
Reddit thread, 2026-09-23: critique, not evidence Demos framed Jev as an agent (cars, games), not a classifier. A commenter: one forward pass caps what it can classify. Another: would old pattern matching do? Jev does not choose its own next action (System One Models) verified. Single-pass ceiling unverified Pitch it as a router in a code-owned loop; baseline rules first (When not to use Jev: rules, embeddings, trained classifiers, small and frontier LLMs)

A simpler tool won

Report What happened Takeaway
@toowitter, unawair v3 Explaining the device logic to Jev simplified the rules until a rule engine could do it; outdoor-temperature response much better Writing the prompt may show plain code suffices: still a win. P24
@gigabit_million, quoted by @satto_sann Rebuilt a Jev classifier in BERT. It worked, but building and updating the criteria is expert work, and satto_sann warns "quick classifier" apps may be a build trap Fixed labels plus a need for transparent criteria → BERT. Else Jev's value is time saved. P34 (inferred)
@voxmenthe, code-search-jev Naive BM25 + Jev wasn't good enough. He added four lexical channels to build candidates, then Jev routes and scores regions: ~2.5 requests, ~18k input tok, ~750 ms warm His ~$0.0015/query contradicts docs arithmetic (18k × $0.042/Mtok ≈ $0.00076; gateway or estimate?). Candidate recall comes first. Patterns: judging, search, documents, real-time and markets P17

Integration and access

Checklist before recommending Jev

  1. Does the answer need generation, counting, date maths, geometry or lookahead? Put that in code and let Jev pick from the candidates (§2–§4, §9).
  2. Is the needed evidence actually in state? Test against a stub that always answers the same.
  3. Do the options cover reality, with "other/none", one judgment each? Shuffle the order in evals.
  4. Faster than reported round trips (260 ms–1 s)? Pause per decision or skip Jev.
  5. Price it: calls/hour × state tokens × $0.042/Mtok, with grouped questions.
  6. Gate on confidence only after calibrating on the project's own labels; Noul for gates.
  7. Baseline rules, BERT, a solver or a small LLM on the same items; use Jev where it wins or ties cheaper.
  8. Never the sole security gate; no trading recommendations.

Sources

Files in frontmatter sources:, captured by 2026-09-23; original URLs inline.