--- title: "Failure reports: confidence misread and calibration" type: community source_tier: community tags: [failure-reports, confidence, calibration, thresholds, community] created: 2026-09-25 updated: 2026-09-25 confidence: medium sources: - raw/x/ranjankumar-2101953564834934999.md - raw/community/primeline-cc-blog-typesafe-jev-pre-registered-test.md - raw/x-repos/tamaratran__fast-jev-compaction.md - raw/x-repos/AbdelStark__jev-benchmarks.md - raw/community/kantahayashiai-github-io-posts-jev-does-not-play-dice.md - raw/x-repos/KantaHayashiAI__jev-does-not-play-dice.md - raw/x-repos/jourdanlabs__assay-001.md - raw/community/donttrustme-ai-assay-001.md - raw/community/agentjournal-dev-blog-llm-judge-vs-feature-extraction.md - raw/x-repos/suraj-phanindra__wellposed.md - raw/x-repos/shitianfang__jev-use__bench-results.md - raw/x-repos/AshutoshVJTI__progressgate.md - raw/x-repos/AshutoshVJTI__progressgate__experiments-readme.md - raw/x-repos/FirasSX914__Janus.md - raw/x-repos/FirasSX914__Janus__research.md - raw/x-repos/collapseindex__jev-ultralightspeed.md - raw/x-repos/collapseindex__jev-ultralightspeed__changelog.md - raw/x-repos/RenaGao__jev-dataops.md - raw/x-repos/RenaGao__jev-dataops__docs-benchmarks.md - raw/x-repos/smkrv__jev-calibrate.md - raw/x-repos/wuyoscar__jev-skill__evals-calibration-results.md - raw/x-repos/fstandhartinger__jevbench__results-v1-4-2.md - raw/x-repos/fstandhartinger__jevbench__docs-method-v1-4.md - raw/community/huggingface-co-datasets-localllama-typed-decisions.md - raw/x-repos/nahid-sparktales__agent-dispatcher__docs-jev.md - raw/x-repos/DanRWilloughby__snifftest__docs-eval-notes.md - raw/x-repos/DanRWilloughby__snifftest.md - raw/x-repos/keltokhy__jsort.md - raw/x-repos/devtooligan__jevscan-evm__bench-batch-stability.md - raw/x-repos/FBddcz__embodied-jev__docs-jev-evaluation.md - raw/x-repos/FBddcz__embodied-jev.md - raw/x-repos/fstandhartinger__jevbench.md - raw/x-repos/shitianfang__jev-use.md - raw/x-repos/wuyoscar__jev-skill.md - raw/x-repos/nahid-sparktales__agent-dispatcher.md - raw/x-repos/devtooligan__jevscan-evm.md - docs/verification/choice-argmax-2026-09-25/README.md jev_version: "jev-1.13.0" summary: "Community reports where Jev's confidence or probabilities misled a gate: overconfidence per dataset, thresholds that don't transfer, confidence vs probability, flat scores from missing evidence." --- # Failure reports: confidence misread and calibration > **TL;DR** `confidence` ranks answers well, but its level is not an accuracy: builders measured Jev overconfident on some datasets (Banking77, Web of Science, a sealed hard set) and calibrated on others, with the best threshold differing by dataset (0.77 to 0.95). Gate only after measuring on your own labels, per question; use a Noul or the option's probability when you need a graded probability; add "other/none". Split from [[ideas/failure-reports]] on 2026-09-25. Marks check against [[concepts/confidence]] and reference/: `confidence` is a statistic of the distribution, not a probability; Noul has no `confidence`; probabilities "sum to approximately 1" ([[reference/http-api]]). Numbers are the builders' own, on `jev-1.13.0`; n is small in most rows. TypeSafe's docs win. ## Confidence misread | Report | Claim | Matches | Takeaway | |---|---|---|---| | [Ranjan Kumar](https://x.com/ranjankumar/status/2101953564834934999) | Trust ordering, not magnitude; a fixed 0.85 gate ignores your prevalence and cost ratio; fit a calibration map on a few hundred own labels | Advice `verified` in direction ([[concepts/confidence]]). "Numbers not trustworthy" `contradicts docs`. No per-customer adaptation `unverified` | Routing/ranking: use argmax. Gating: calibrate on the project's own data first. [[ideas/patterns-agents]] P06 | | primeline.cc | ≥0.9 confidence: ~92% right on 73% of items; below 0.8 about 50%. Calibration error by type: Noul 0.012, Choice 0.086, Score 0.254 (slightly overconfident) | Noul has no `confidence` field: `verified` ([[reference/openapi-schemas]]). CJK caveat credited to the jaggedness doc: `contradicts docs` (not there) | Prefer Noul for gates. Never reuse a threshold across question types (§8) | | Same article ([primeline.cc](https://primeline.cc/blog/typesafe-jev-pre-registered-test)): yelban replaying [fast-jev-compaction](https://github.com/tamaratran/fast-jev-compaction) | None of 256 "keep" scores reached 0.3. A stub that always answers zero gave the same outcome (87.7% vs 88.5% character reduction; 240 vs 244 results dropped or truncated). The plugin sent only the result's length and tool name, never the text | §5 in spirit `verified`: the content was missing | Flat low scores usually mean `state` lacks the evidence; compare against a stub. P07 | | [jev-benchmarks](https://github.com/AbdelStark/jev-benchmarks) (AbdelStark), pre-registered pilot vs local GLiNER2.5, 100 items per set | Jev ahead on AG News, Banking77 (0.910, 0.870 vs 0.700, 0.610). 6-label emotion: accuracy tie (0.480 vs 0.440), Brier 0.846 vs 0.668, true label at probability 0 for 16% | Docs call probabilities calibrated ([[concepts/confidence]]): per-task exception, n=100, `unverified` | Check calibration per label set before gating; fuzzy labels can be confidently wrong | | [Jev does not play dice](https://kantahayashiai.github.io/posts/jev-does-not-play-dice/), Kanta Hayashi ([repo](https://github.com/KantaHayashiAI/jev-does-not-play-dice)), via Vercel AI Gateway | Hidden fair die, Choice over 6 faces, 400 trials: always "1", mean probability 82.9%, right 19.0%. Fair coin 92.0% vs 52.0%. Noul "did it show 1?" 19.2% (true 16.7%) but 15.0% for a true 5.0% with 20 options. A document stating 45% vs 55% risk came back 6.6% vs 95.9% | Docs define calibration this way ([[concepts/machine-learning-primer]]): per-prompt exception, `unverified` in general (the author says so) | Test unknowable cases before gating; never let a Choice overwrite a probability stated in the source, extract it instead | | [ASSAY-001](https://donttrustme.ai/assay-001.html), JourdanLabs, pre-registered | Banking77 (n = 3,080): ECE 0.0936, overconfident in every bin below 0.9 (says 0.86, right 67%; says 0.55, right 36%). CLINC150: ECE 0.0204, calibrated. 0 type errors in 8,576 responses | Per-task exception to the docs' calibration claim: `unverified` in general, measured here | Calibration is per dataset; re-measure on yours. [[ideas/head-to-head-benchmarks]] | | [agentjournal](https://agentjournal.dev/blog/llm-judge-vs-feature-extraction/), ikkun: hard synthetic task, 300 rows | The direct call reported ≥ 0.9 confidence on 126 rows (42%) and was right on only 72.2% of them | `unverified` in general | Hard, weak-cue tasks break the high-confidence band too | | [wellposed](https://github.com/suraj-phanindra/wellposed), suraj-phanindra | A Choice with no catch-all option answered a wrong option at confidence **1.00**; with an "other" option it picked "other" at 0.93 | Fix matches [[concepts/choice]] ("other/none" option) `verified`; the 1.00 figure `unverified` | Confidence gates cannot catch a missing option: add "other/none" | | [jev-use](https://github.com/shitianfang/jev-use), shitianfang: 3-option paddle question, 80 calls | Never answered `stay`: right on every move state, 0 of 13 hold states | `unverified` | An option the model never selects fails silently: check the answer distribution per option | | [ProgressGate](https://github.com/AshutoshVJTI/progressgate), AshutoshVJTI: six trajectory Nouls | `materialProgress` scored 0.42 on a step whose result read "deploy succeeded"; it lags after stagnant steps. Automatic HALT is off by default | `unverified`; tested on 53 hand-labelled + 50 adversarial trajectories | Use such a signal as a veto inside a policy, not alone | | [Janus](https://github.com/FirasSX914/Janus) (FirasSX914): Banking77 and Web of Science (145 classes), 500 each | Overconfident at every level: mean confidence 90.7% vs accuracy 77.8% (Banking77), 83.2% vs 52.8% (WoS); 10-bin ECE 0.1302 / 0.3047; Brier 0.3518 / 0.7491. Confidence 1.00 on 47.6% of Banking77 at 95.8% right, 25.8% of WoS at 76.7%. No margin or entropy variant beat `confidence` on AUROC. 33 of 500 rows' probabilities sum to 0.99 | Sums of 0.99 fit "approximately 1" `verified`. Overconfidence per dataset `unverified` in general; ASSAY-001 above measured Banking77 ECE 0.0936 on 3,080 (different sample and prompt) | Ranking holds, level does not. Its cascade result: [[ideas/head-to-head]] | | [jev-ultralightspeed](https://github.com/collapseindex/jev-ultralightspeed) (collapseindex): XSTest 1,347 human-labelled, BoolQ and AG News | Thresholds did not transfer: best cut 0.92 on XSTest, 0.77 BoolQ, 0.95 AG News. Overconfident on XSTest and AG News, underconfident on BoolQ; mid-range it said 74.8% and was right 51.0%. Holding out the least-sure 19% raised agreement 89.7% → 96.8%. A six-vote majority scored 89.5% vs 89.7% for one ask | `unverified` in general | One threshold per question and dataset; repeating the same ask buys nothing. Packing and trimming results: [[ideas/failure-reports]], [[ideas/request-mechanics]] | | [jev-dataops](https://github.com/RenaGao/jev-dataops) (RenaGao): keep / review / reject for training rows | `confidence` sat near 0.5 on a 3-way `quality` question while P(good) was 0.7-0.9, so a confidence ≥ 0.85 gate sent 80% of a clean 84-row set to review (15 / 67 / 2). Gating on the chosen option's probability: 60 / 21 / 3. On 63 labelled rows, clean rows kept rose 18/45 → 37/45 with all 18 bad rows still rejected | Confidence ≠ probability `verified` ([[concepts/confidence]]); the 0.5 figure `unverified`. Small synthetic sets | For a gate, threshold the option's probability; a `confidence` cut built for one option count misfires on another | | [jev-calibrate](https://github.com/smkrv/jev-calibrate) (smkrv): bundled 66-message example | First `frustration` draft 18/26; 4 of its 8 misses had confidence ≥ 0.94, so a 0.9 cut still gave 0.76. Rewriting the `criteria` took it 0.69 → 0.92 (holdout AUC 1.00 / 1.00 / 0.97) | "Noul has no `confidence`" `verified`. Easy set; one person wrote questions and labels | Confident misses come from wording; fix the criteria before tuning a threshold ([[ideas/patterns-emerging]] P42) | | [jev-skill](https://github.com/wuyoscar/jev-skill) (wuyoscar): 160 BBH items, 4 tasks | 136/160 vs DeepSeek 120/160 (strict interface). The ≥ 0.90 band was 92/100 with 8 wrong, 6 of them causal judgment (14/20 there). Top-probability bin [.9,1]: 118 items, mean 0.9814, right 0.8898. Middle bands not monotonic ([.70,.90) 70.4%, below .70 75.8%). Top probability ≥ .90 accepts 118 items, `confidence` ≥ .90 accepts 100 | `unverified` in general; the author calls the sample small and mixed | Thresholds are per task; top probability and `confidence` are different signals, don't swap them silently | | JevBench v1.4.2 sealed set ([method](https://github.com/fstandhartinger/jevbench), fstandhartinger): 308 hard items unseen by builders | Jev 36.7% accuracy (86.6% on the public items; chance 29.3%). Sealed ECE 0.220 vs 0.061 on the public hard tier. Confidence ≥ 0.9 covered 8.4% of items at 42.3% right; ≥ 0.7 covered 29.2% at 40.0%. Row flagged API: TypeSafe's endpoint saw the sealed item text, without answers | `unverified` in general; the set is built to be hard for one-pass models | On hard, unfamiliar items the high-confidence band stops meaning "right". Board detail: [[ideas/eval-boards]] | | [typed-decisions card](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) (LocalLLaMA): 400 cases, 2,000 decisions, run live 2026-09-18 | Accuracy 0.727 against a teacher ceiling of 0.735, but KL from the gold distributions 1.442 vs 0.223 for a fitted ModernBERT; ECE 0.144 (overconfidence +0.023). Gold is the mean of three samples from an unnamed ~4B teacher, so accuracy here is agreement with that teacher | `unverified` in general | Jev commits to one answer; if you use its distribution as a soft label, it will not reproduce a spread of opinions | | [agent-dispatcher](https://github.com/nahid-sparktales/agent-dispatcher) (nahid-sparktales): 162 routing fixtures | Accuracy fell to 0.69 in the 0.7-0.8 confidence band | `unverified` | Route the middle band elsewhere. Full comparison: [[ideas/head-to-head-agents]] | | [Sniff Test](https://github.com/DanRWilloughby/snifftest) (DanRWilloughby): one Noul per rule per paragraph | Readings near 0.5 came back on text Jev could not read, so the tool treats 0.4-0.6 as "no judgment". Same-day reruns caught 59-64 of 80 faults | `unverified` | A reading near 0.5 means "cannot tell", not "half likely" | | [jsort](https://github.com/keltokhy/jsort) (keltokhy): Choice vs Noul probe | Both 95.6% correct, but a 2-option Choice pushed 93% of answers below 0.1 or above 0.9 where the Noul left 28% in between | `unverified` | For a graded probability ask a Noul, not a 2-option Choice | | [jevscan-evm](https://github.com/devtooligan/jevscan-evm) (devtooligan): 20 repeated pairs | Repeat noise mean 0.009, max 0.03; batching shifted scores about as much; single swings of 0.05-0.08. Author: p within ~0.05 of a threshold is a tie | `unverified` | Treat near-threshold scores as ties; batching evidence both ways: [[ideas/request-mechanics]] | | `choice` not the top probability: one fixture in [embodied-jev](https://github.com/FBddcz/embodied-jev) (FBddcz) | One `jev-1.13.0` answer had `choice` `y_neg_40` at 0.18 while `z_neg_40` was 0.19 (18% vs 19% in the author's docs/JEV_EVALUATION.md) | `contradicts docs` (`choice` is the highest-probability option, [[reference/http-api]]). Our 2026-09-25 test: 100/100 Choice answers had `choice` = highest probability via OpenCode Zen's free `jev-1.13-free` route (same weights as `jev-1.13.0` `unverified`); not reproduced | Read `probabilities` when a near-tie matters; log mismatches | ## How to use confidence safely 1. Rank or route by the answer; gate only after measuring accuracy per confidence band on your own labels. 2. Set one threshold per question, option count and dataset; they did not transfer (0.77 to 0.95 on three datasets). 3. For a yes/no gate use a Noul (no `confidence`; its probability is the number to threshold). For a 3-way gate, threshold the option's probability. 4. Add an "other/none" option; check that every option is ever chosen. 5. Test cases the `state` cannot answer (dice, missing evidence): Jev still answers, often confidently. 6. Treat readings within ~0.05 of a threshold, or near 0.5 on a Noul, as "cannot tell". ## Related - [[ideas/failure-reports]]: every other failure kind and the pre-recommendation checklist - [[concepts/confidence]]: the official definition; [[patterns/confidence-routing]]: the official cascade - [[ideas/head-to-head-benchmarks]]: calibration on public sets; [[ideas/request-mechanics]]: calibration and stability probes - [[ideas/patterns-agents]] P06, [[ideas/patterns-emerging]] P42 ## Sources Files in frontmatter `sources:`, captured by 2026-09-25; original URLs inline. The `choice` = argmax test is ours: docs/verification/choice-argmax-2026-09-25/README.md (repo only).