Agents: read the raw Markdown of this page, or start at llms.txt.
~/wiki/ideas
Head-to-head benchmarks: security (phishing, spam, injection, vulnerable code)
TL;DR On public security sets Jev ranks injections well (AUC 0.97 or more on deepset, InjecAgent and BIPIA email in four builders' runs) and ties trained spam classifiers, but it lost a one-question phishing verdict to Haiku, a two-rule regex beat its best single phishing signal, it missed most BIPIA EmailQA injections with one wording, and its accuracy at a fixed cut depends on telling it what the deployment is. Write the regex or rule baseline first, describe the deployment in
state, and mark tool results untrusted. Split from Head-to-head benchmarks: Jev on public datasets and suites on 2026-09-25; rows moved unchanged except the AY Automate row's cross-reference.
Numbers as each builder published them, on Jev 1.13 (jev-1.13.0) unless stated; rows that name no version are assumed on it (inferred: jev-latest pointed at jev-1.13.0 throughout 2026-09, Models, aliases, pricing, rate limits, context). LOST = Jev lost on the stated metric. Public sets can be in any model's pretraining; one run each. Adversarial input is a documented failure mode (Jev 1.13 jaggedness: known failure modes §6).
Phishing, spam, injection and vulnerable code
| Dataset (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| PhishNChips v5.2, 2,000 emails (anisselbd/jev-phishing-bench; the study Aman Kumar relays) | Jev verdict Choice vs Claude Haiku 4.5; then 5 signal Nouls + regression; controls added after review | Verdict 62.6% vs 81.3% (McNemar p < 0.0001; phishing recall 43.2% vs 76.4%, false positives 18.0% vs 13.8%); ECE 0.154 vs 0.097; p50 239 vs 687 ms; $0.038 vs $0.462 per 1,000. On a held-out half: Jev's five signals + regression 95.0%, beating the regex (91.8%, p = 0.002); Haiku asked the same five questions 93.2% (p = 0.063, tie; Haiku AUROC higher); a two-rule regex 91.8% beats Jev's best single signal 89.4% (p = 0.003) | LOST verdict; signals tie Haiku at ~27× less cost | Synthetic bodies, labels from URL feeds, signals written after reading the dataset's URL taxonomy; the regex alone scores 91.6% on all 2,000; the author says the dataset "largely separates by construction". No licence (describe only) |
| Ham / spam / phishing, 9,886 messages (bitnovus/jev-spam-eval) | Jev zero-shot Choice vs TF-IDF logistic regression | Main test (5,733): 98.64% (enriched, extra wording) vs 98.87%. Adding link targets, Reply-To and attachment metadata to the state, same question, gave the largest gain (Builder lessons: what changed the result). 853 phishing mails from 2024-25: 95.31% vs 75.26% recall. 50/50 average: 99.30%. Earlier binary run, 18,514 unique messages (five-fold CV, near-duplicate groups kept together): Jev detailed criteria 98.33% vs TF-IDF regression 98.39% | Tie on main; recall on recent phishing | Wording informed by earlier labelled errors; regression kept the lead on fresh mail without near-copies: 95.08% vs enriched Jev 93.83% (original question), 94.33% with the extra wording; the fixed 50/50 average did not beat regression on fresh mail (96.21% vs 96.33%); recent set has no legitimate mail |
| Spam, 19,500 emails (relayed by Kumar) | vs logistic regression trained on 15,000 | 98.3% zero-shot, level | Tie with no training | Original not captured. bitnovus's earlier binary run (its row above) may be this study (inferred); kept apart |
| deepset/prompt-injections, all 662 (jev-sec-bench, Gaurav Gosain) | Same Noul with vs without a line saying what the assistant is for (a news publisher's reader assistant) | 2026-09-16, one request per message, per-sample results published. At a plain 0.50 cut: 96.5% vs 89.7%, recall 95.1% vs 74.9%, ROC-AUC 0.9927 vs 0.9846, ECE 0.0588 vs 0.0928. Without the line it caught blatant overrides (precision 99.0%) but read topic hijacks as ordinary requests | Context line wins | Under-confident at the top (the 0.85 bucket right 44/44). Vulnerable-code pairs: vulnerable half scored above its secure twin in 178/200, absolute accuracy 71.5%; ≥ 38% of false positives are corpus label errors per the author (unverified) |
| deepset/prompt-injections, 400 of 662 (AY Automate, agency; its Banking77 row: Head-to-head benchmarks: Jev on public datasets and suites) | vs GPT-5.4 nano, Gemini 3.5 Flash-Lite, Haiku 4.5, GPT-5.6 Terra, via OpenRouter | Jev AUROC 0.990 (best of five); at 0.50 recall 0.69, precision 1.00; at 0.10 recall 0.96, precision 0.92 | Ranking; beat nano, tied the rest on accuracy | Cutoffs picked after seeing results; dataset mixes English and German, labels partly persona prompts |
| Hard-benign texts that mention attacks (339) and hackaprompt (1,383) (agentjournal) | One direct Choice vs 12 Jev dimensions + weights vs word-bigram NB | Hard benigns false-positive rate 1.5% vs 37.2% vs 43.4%; hackaprompt 0.665 vs 0.977 vs 0.994; external set 0.956 vs 0.840 vs 0.381 | Direct question on benign text; dimensions on attack strings | Author's own dimension design; four repair attempts failed |
| InjecAgent (1,394), BIPIA email (250), 100 hand-labelled tool calls (agent-chaperone, agent-chaperone; 2026-09-21, 1,947 requests, $0.062) | A Jev question battery screening tool calls and tool results | InjecAgent AUC 0.976 (precision / recall 0.989 / 0.805 at 0.5); BIPIA email AUC 1.000, recall 0.825 at 0.5; hand-labelled calls AUC 0.993; 68 benign texts about injection gave 8 false positives at 0.5 (bench README: 7) | Ranking | Raw responses published. Policy-violation, off-task and tool-description questions unmeasured |
| deepset, LLMail, BIPIA EmailQA, held-out tool results (jev-edge, kiwi0719) | A live Jev check behind cheap rules in an OpenResty gateway | deepset AUC 0.983 with 37.3% missed; 0.996 and 5.3% missed once the deployment is described (as jev-sec-bench above). LLMail 29.8% missed; BIPIA EmailQA 81.5% missed; tool results 77.8% → 19.0% missed when marked untrusted | Context line wins | One run each. Misses match Jev 1.13 jaggedness: known failure modes §6: verified in direction. BIPIA results conflict with agent-chaperone's (BIPIA email recall 0.825 at 0.5 vs 81.5% missed): different question wording and BIPIA task; cite both |
| Judged audit contests USSD, Beedle, Monolith (jevscan-evm, devtooligan) | 359 Nouls per Solidity file (2 general + 14 category + 343 checklist-derived) vs contest findings; vs GPT-5.6 high on one file, 14 checks | Findings located at file level ≥ 70%: USSD 20/22 (0 false positives), Beedle 27/42 (1), Monolith 6/7 (3); function level 16, 18, 5. One file, 14 checks: 0.7 s and 0.015¢ vs 57 s and 3.9¢ | Triage heatmap | Author says the code is vibe-coded and unread. Batch stability: Failure reports: confidence misread and calibration |
How to read these
- A deployment line moves the cut, not the ranking (lesson: Builder lessons: what changed the result): jev-sec-bench and jev-edge both gained most once the assistant's purpose was in
state; AUC barely moved. - Wording and task decide injection recall: jev-edge and agent-chaperone disagree on BIPIA email; cite both.
- Simple baselines first: a two-rule regex beat Jev's best single phishing signal; trained regression matched Jev on spam with labels and a stable corpus.
- Patterns: Patterns: judging, search, documents, real-time and markets P16 screening; Patterns: agent internals, routing, gates, context and memory P03 safety gates; tools: Tools: guardrails for agents (tool-call gates, permission hooks, injection screens, rule checks).
Related
- Head-to-head benchmarks: Jev on public datasets and suites: the other public sets (intent, retrieval, science, robots, reasoning); Head-to-head: Jev against other models and methods: app-level comparisons; Eval boards: JevBench by version, Jevals.com, typed-decisions and other independent suites: boards
- Failure reports: where Jev broke, lost, or was the wrong tool, Failure reports: confidence misread and calibration: where these broke; Cost ledger: published cost per Jev decision: cost per decision
- Jev 1.13 jaggedness: known failure modes, Confidence vs probability
Sources
Links inline; raw captures listed in the frontmatter (private repo).