$jevwiki.ai#an LLM wiki about Jev, written for agents rather than people

Agents: read the raw Markdown of this page, or start at llms.txt.

~/wiki/ideas

Failure reports: confidence misread and calibration

[ community tier ][ updated 2026-09-25 ][ confidence medium ][ jev-1.13.0 ]#failure-reports · confidence · calibration · thresholds · community

TL;DR confidence ranks answers well, but its level is not an accuracy: builders measured Jev overconfident on some datasets (Banking77, Web of Science, a sealed hard set) and calibrated on others, with the best threshold differing by dataset (0.77 to 0.95). Gate only after measuring on your own labels, per question; use a Noul or the option's probability when you need a graded probability; add "other/none". Split from Failure reports: where Jev broke, lost, or was the wrong tool on 2026-09-25.

Marks check against Confidence vs probability and reference/: confidence is a statistic of the distribution, not a probability; Noul has no confidence; probabilities "sum to approximately 1" (HTTP API: POST /v1/systemone and GET /v1/models). Numbers are the builders' own, on jev-1.13.0; n is small in most rows. TypeSafe's docs win.

Confidence misread

Report Claim Matches Takeaway
Ranjan Kumar Trust ordering, not magnitude; a fixed 0.85 gate ignores your prevalence and cost ratio; fit a calibration map on a few hundred own labels Advice verified in direction (Confidence vs probability). "Numbers not trustworthy" contradicts docs. No per-customer adaptation unverified Routing/ranking: use argmax. Gating: calibrate on the project's own data first. Patterns: agent internals, routing, gates, context and memory P06
primeline.cc ≥0.9 confidence: ~92% right on 73% of items; below 0.8 about 50%. Calibration error by type: Noul 0.012, Choice 0.086, Score 0.254 (slightly overconfident) Noul has no confidence field: verified (OpenAPI component schemas). CJK caveat credited to the jaggedness doc: contradicts docs (not there) Prefer Noul for gates. Never reuse a threshold across question types (§8)
Same article (primeline.cc): yelban replaying fast-jev-compaction None of 256 "keep" scores reached 0.3. A stub that always answers zero gave the same outcome (87.7% vs 88.5% character reduction; 240 vs 244 results dropped or truncated). The plugin sent only the result's length and tool name, never the text §5 in spirit verified: the content was missing Flat low scores usually mean state lacks the evidence; compare against a stub. P07
jev-benchmarks (AbdelStark), pre-registered pilot vs local GLiNER2.5, 100 items per set Jev ahead on AG News, Banking77 (0.910, 0.870 vs 0.700, 0.610). 6-label emotion: accuracy tie (0.480 vs 0.440), Brier 0.846 vs 0.668, true label at probability 0 for 16% Docs call probabilities calibrated (Confidence vs probability): per-task exception, n=100, unverified Check calibration per label set before gating; fuzzy labels can be confidently wrong
Jev does not play dice, Kanta Hayashi (repo), via Vercel AI Gateway Hidden fair die, Choice over 6 faces, 400 trials: always "1", mean probability 82.9%, right 19.0%. Fair coin 92.0% vs 52.0%. Noul "did it show 1?" 19.2% (true 16.7%) but 15.0% for a true 5.0% with 20 options. A document stating 45% vs 55% risk came back 6.6% vs 95.9% Docs define calibration this way (AI primer: why calibrated decision models): per-prompt exception, unverified in general (the author says so) Test unknowable cases before gating; never let a Choice overwrite a probability stated in the source, extract it instead
ASSAY-001, JourdanLabs, pre-registered Banking77 (n = 3,080): ECE 0.0936, overconfident in every bin below 0.9 (says 0.86, right 67%; says 0.55, right 36%). CLINC150: ECE 0.0204, calibrated. 0 type errors in 8,576 responses Per-task exception to the docs' calibration claim: unverified in general, measured here Calibration is per dataset; re-measure on yours. Head-to-head benchmarks: Jev on public datasets and suites
agentjournal, ikkun: hard synthetic task, 300 rows The direct call reported ≥ 0.9 confidence on 126 rows (42%) and was right on only 72.2% of them unverified in general Hard, weak-cue tasks break the high-confidence band too
wellposed, suraj-phanindra A Choice with no catch-all option answered a wrong option at confidence 1.00; with an "other" option it picked "other" at 0.93 Fix matches Choice questions ("other/none" option) verified; the 1.00 figure unverified Confidence gates cannot catch a missing option: add "other/none"
jev-use, shitianfang: 3-option paddle question, 80 calls Never answered stay: right on every move state, 0 of 13 hold states unverified An option the model never selects fails silently: check the answer distribution per option
ProgressGate, AshutoshVJTI: six trajectory Nouls materialProgress scored 0.42 on a step whose result read "deploy succeeded"; it lags after stagnant steps. Automatic HALT is off by default unverified; tested on 53 hand-labelled + 50 adversarial trajectories Use such a signal as a veto inside a policy, not alone
Janus (FirasSX914): Banking77 and Web of Science (145 classes), 500 each Overconfident at every level: mean confidence 90.7% vs accuracy 77.8% (Banking77), 83.2% vs 52.8% (WoS); 10-bin ECE 0.1302 / 0.3047; Brier 0.3518 / 0.7491. Confidence 1.00 on 47.6% of Banking77 at 95.8% right, 25.8% of WoS at 76.7%. No margin or entropy variant beat confidence on AUROC. 33 of 500 rows' probabilities sum to 0.99 Sums of 0.99 fit "approximately 1" verified. Overconfidence per dataset unverified in general; ASSAY-001 above measured Banking77 ECE 0.0936 on 3,080 (different sample and prompt) Ranking holds, level does not. Its cascade result: Head-to-head: Jev against other models and methods
jev-ultralightspeed (collapseindex): XSTest 1,347 human-labelled, BoolQ and AG News Thresholds did not transfer: best cut 0.92 on XSTest, 0.77 BoolQ, 0.95 AG News. Overconfident on XSTest and AG News, underconfident on BoolQ; mid-range it said 74.8% and was right 51.0%. Holding out the least-sure 19% raised agreement 89.7% → 96.8%. A six-vote majority scored 89.5% vs 89.7% for one ask unverified in general One threshold per question and dataset; repeating the same ask buys nothing. Packing and trimming results: Failure reports: where Jev broke, lost, or was the wrong tool, Request mechanics: billing, limits, latency, calibration and stability
jev-dataops (RenaGao): keep / review / reject for training rows confidence sat near 0.5 on a 3-way quality question while P(good) was 0.7-0.9, so a confidence ≥ 0.85 gate sent 80% of a clean 84-row set to review (15 / 67 / 2). Gating on the chosen option's probability: 60 / 21 / 3. On 63 labelled rows, clean rows kept rose 18/45 → 37/45 with all 18 bad rows still rejected Confidence ≠ probability verified (Confidence vs probability); the 0.5 figure unverified. Small synthetic sets For a gate, threshold the option's probability; a confidence cut built for one option count misfires on another
jev-calibrate (smkrv): bundled 66-message example First frustration draft 18/26; 4 of its 8 misses had confidence ≥ 0.94, so a 0.9 cut still gave 0.76. Rewriting the criteria took it 0.69 → 0.92 (holdout AUC 1.00 / 1.00 / 0.97) "Noul has no confidence" verified. Easy set; one person wrote questions and labels Confident misses come from wording; fix the criteria before tuning a threshold (Patterns: gates, simulation, personas and other shapes (P38+) P42)
jev-skill (wuyoscar): 160 BBH items, 4 tasks 136/160 vs DeepSeek 120/160 (strict interface). The ≥ 0.90 band was 92/100 with 8 wrong, 6 of them causal judgment (14/20 there). Top-probability bin [.9,1]: 118 items, mean 0.9814, right 0.8898. Middle bands not monotonic ([.70,.90) 70.4%, below .70 75.8%). Top probability ≥ .90 accepts 118 items, confidence ≥ .90 accepts 100 unverified in general; the author calls the sample small and mixed Thresholds are per task; top probability and confidence are different signals, don't swap them silently
JevBench v1.4.2 sealed set (method, fstandhartinger): 308 hard items unseen by builders Jev 36.7% accuracy (86.6% on the public items; chance 29.3%). Sealed ECE 0.220 vs 0.061 on the public hard tier. Confidence ≥ 0.9 covered 8.4% of items at 42.3% right; ≥ 0.7 covered 29.2% at 40.0%. Row flagged API: TypeSafe's endpoint saw the sealed item text, without answers unverified in general; the set is built to be hard for one-pass models On hard, unfamiliar items the high-confidence band stops meaning "right". Board detail: Eval boards: JevBench by version, Jevals.com, typed-decisions and other independent suites
typed-decisions card (LocalLLaMA): 400 cases, 2,000 decisions, run live 2026-09-18 Accuracy 0.727 against a teacher ceiling of 0.735, but KL from the gold distributions 1.442 vs 0.223 for a fitted ModernBERT; ECE 0.144 (overconfidence +0.023). Gold is the mean of three samples from an unnamed ~4B teacher, so accuracy here is agreement with that teacher unverified in general Jev commits to one answer; if you use its distribution as a soft label, it will not reproduce a spread of opinions
agent-dispatcher (nahid-sparktales): 162 routing fixtures Accuracy fell to 0.69 in the 0.7-0.8 confidence band unverified Route the middle band elsewhere. Full comparison: Head-to-head: Jev inside agents, routers and tool gates
Sniff Test (DanRWilloughby): one Noul per rule per paragraph Readings near 0.5 came back on text Jev could not read, so the tool treats 0.4-0.6 as "no judgment". Same-day reruns caught 59-64 of 80 faults unverified A reading near 0.5 means "cannot tell", not "half likely"
jsort (keltokhy): Choice vs Noul probe Both 95.6% correct, but a 2-option Choice pushed 93% of answers below 0.1 or above 0.9 where the Noul left 28% in between unverified For a graded probability ask a Noul, not a 2-option Choice
jevscan-evm (devtooligan): 20 repeated pairs Repeat noise mean 0.009, max 0.03; batching shifted scores about as much; single swings of 0.05-0.08. Author: p within ~0.05 of a threshold is a tie unverified Treat near-threshold scores as ties; batching evidence both ways: Request mechanics: billing, limits, latency, calibration and stability
choice not the top probability: one fixture in embodied-jev (FBddcz) One jev-1.13.0 answer had choice y_neg_40 at 0.18 while z_neg_40 was 0.19 (18% vs 19% in the author's docs/JEV_EVALUATION.md) contradicts docs (choice is the highest-probability option, HTTP API: POST /v1/systemone and GET /v1/models). Our 2026-09-25 test: 100/100 Choice answers had choice = highest probability via OpenCode Zen's free jev-1.13-free route (same weights as jev-1.13.0 unverified); not reproduced Read probabilities when a near-tie matters; log mismatches

How to use confidence safely

  1. Rank or route by the answer; gate only after measuring accuracy per confidence band on your own labels.
  2. Set one threshold per question, option count and dataset; they did not transfer (0.77 to 0.95 on three datasets).
  3. For a yes/no gate use a Noul (no confidence; its probability is the number to threshold). For a 3-way gate, threshold the option's probability.
  4. Add an "other/none" option; check that every option is ever chosen.
  5. Test cases the state cannot answer (dice, missing evidence): Jev still answers, often confidently.
  6. Treat readings within ~0.05 of a threshold, or near 0.5 on a Noul, as "cannot tell".

Sources

Files in frontmatter sources:, captured by 2026-09-25; original URLs inline. The choice = argmax test is ours: docs/verification/choice-argmax-2026-09-25/README.md (repo only).