---
title: "Head-to-head benchmarks: security (phishing, spam, injection, vulnerable code)"
type: community
source_tier: community
tags: [head-to-head, benchmarks, security, prompt-injection, community]
created: 2026-09-25
updated: 2026-09-25
confidence: medium
sources:
  - raw/x-repos/anisselbd__jev-phishing-bench.md
  - raw/community/amankumar-ai-blogs-jev-measured.md
  - raw/x-repos/bitnovus__jev-spam-eval.md
  - raw/x-repos/Gaurav-Gosain__jev-sec-bench.md
  - raw/community/ayautomate-com-blog-jev-vs-llm-benchmark.md
  - raw/community/agentjournal-dev-blog-llm-judge-vs-feature-extraction.md
  - raw/x-repos/agent-chaperone__agent-chaperone.md
  - raw/x-repos/agent-chaperone__agent-chaperone__bench-readme.md
  - raw/x-repos/kiwi0719__jev-edge.md
  - raw/x-repos/kiwi0719__jev-edge__bench-report.md
  - raw/x-repos/devtooligan__jevscan-evm.md
jev_version: "jev-1.13.0"
summary: "Builder-run Jev scores on public security sets: PhishNChips phishing, spam corpora, deepset and BIPIA prompt injection, InjecAgent, audit contests. Split from head-to-head-benchmarks."
---

# Head-to-head benchmarks: security (phishing, spam, injection, vulnerable code)

> **TL;DR** On public security sets Jev ranks injections well (AUC 0.97 or more on deepset, InjecAgent and BIPIA email in four builders' runs) and ties trained spam classifiers, but it lost a one-question phishing verdict to Haiku, a two-rule regex beat its best single phishing signal, it missed most BIPIA EmailQA injections with one wording, and its accuracy at a fixed cut depends on telling it what the deployment is. Write the regex or rule baseline first, describe the deployment in `state`, and mark tool results untrusted. Split from [[ideas/head-to-head-benchmarks]] on 2026-09-25; rows moved unchanged except the AY Automate row's cross-reference.

Numbers as each builder published them, on Jev 1.13 (`jev-1.13.0`) unless stated; rows that name no version are assumed on it (inferred: `jev-latest` pointed at `jev-1.13.0` throughout 2026-09, [[reference/models-and-pricing]]). **LOST** = Jev lost on the stated metric. Public sets can be in any model's pretraining; one run each. Adversarial input is a documented failure mode ([[concepts/jaggedness-jev-1-13]] §6).

## Phishing, spam, injection and vulnerable code

| Dataset (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| PhishNChips v5.2, 2,000 emails ([anisselbd/jev-phishing-bench](https://github.com/anisselbd/jev-phishing-bench); the study [Aman Kumar](https://amankumar.ai/blogs/jev-measured) relays) | Jev verdict Choice vs Claude Haiku 4.5; then 5 signal Nouls + regression; controls added after review | Verdict 62.6% vs 81.3% (McNemar p < 0.0001; phishing recall 43.2% vs 76.4%, false positives 18.0% vs 13.8%); ECE 0.154 vs 0.097; p50 239 vs 687 ms; $0.038 vs $0.462 per 1,000. On a held-out half: Jev's five signals + regression 95.0%, beating the regex (91.8%, p = 0.002); Haiku asked the same five questions 93.2% (p = 0.063, tie; Haiku AUROC higher); a two-rule regex 91.8% **beats** Jev's best single signal 89.4% (p = 0.003) | **LOST** verdict; signals tie Haiku at ~27× less cost | Synthetic bodies, labels from URL feeds, signals written after reading the dataset's URL taxonomy; the regex alone scores 91.6% on all 2,000; the author says the dataset "largely separates by construction". No licence (describe only) |
| Ham / spam / phishing, 9,886 messages ([bitnovus/jev-spam-eval](https://github.com/bitnovus/jev-spam-eval)) | Jev zero-shot Choice vs TF-IDF logistic regression | Main test (5,733): 98.64% (enriched, extra wording) vs 98.87%. Adding link targets, Reply-To and attachment metadata to the state, same question, gave the largest gain ([[ideas/builder-lessons]]). 853 phishing mails from 2024-25: 95.31% vs 75.26% recall. 50/50 average: 99.30%. Earlier binary run, 18,514 unique messages (five-fold CV, near-duplicate groups kept together): Jev detailed criteria 98.33% vs TF-IDF regression 98.39% | Tie on main; recall on recent phishing | Wording informed by earlier labelled errors; regression kept the lead on fresh mail without near-copies: 95.08% vs enriched Jev 93.83% (original question), 94.33% with the extra wording; the fixed 50/50 average did not beat regression on fresh mail (96.21% vs 96.33%); recent set has no legitimate mail |
| Spam, 19,500 emails (relayed by Kumar) | vs logistic regression trained on 15,000 | 98.3% zero-shot, level | Tie with no training | Original not captured. bitnovus's earlier binary run (its row above) may be this study (inferred); kept apart |
| deepset/prompt-injections, all 662 ([jev-sec-bench](https://github.com/Gaurav-Gosain/jev-sec-bench), Gaurav Gosain) | Same Noul with vs without a line saying what the assistant is for (a news publisher's reader assistant) | 2026-09-16, one request per message, per-sample results published. At a plain 0.50 cut: 96.5% vs 89.7%, recall 95.1% vs 74.9%, ROC-AUC 0.9927 vs 0.9846, ECE 0.0588 vs 0.0928. Without the line it caught blatant overrides (precision 99.0%) but read topic hijacks as ordinary requests | Context line wins | Under-confident at the top (the 0.85 bucket right 44/44). Vulnerable-code pairs: vulnerable half scored above its secure twin in 178/200, absolute accuracy 71.5%; ≥ 38% of false positives are corpus label errors per the author (`unverified`) |
| deepset/prompt-injections, 400 of 662 ([AY Automate](https://www.ayautomate.com/blog/jev-vs-llm-benchmark), agency; its Banking77 row: [[ideas/head-to-head-benchmarks]]) | vs GPT-5.4 nano, Gemini 3.5 Flash-Lite, Haiku 4.5, GPT-5.6 Terra, via OpenRouter | Jev AUROC **0.990** (best of five); at 0.50 recall 0.69, precision 1.00; at 0.10 recall 0.96, precision 0.92 | Ranking; beat nano, tied the rest on accuracy | Cutoffs picked after seeing results; dataset mixes English and German, labels partly persona prompts |
| Hard-benign texts that mention attacks (339) and hackaprompt (1,383) ([agentjournal](https://agentjournal.dev/blog/llm-judge-vs-feature-extraction/)) | One direct Choice vs 12 Jev dimensions + weights vs word-bigram NB | Hard benigns false-positive rate 1.5% vs **37.2%** vs 43.4%; hackaprompt 0.665 vs 0.977 vs 0.994; external set 0.956 vs 0.840 vs 0.381 | Direct question on benign text; dimensions on attack strings | Author's own dimension design; four repair attempts failed |
| InjecAgent (1,394), BIPIA email (250), 100 hand-labelled tool calls ([agent-chaperone](https://github.com/agent-chaperone/agent-chaperone), agent-chaperone; 2026-09-21, 1,947 requests, $0.062) | A Jev question battery screening tool calls and tool results | InjecAgent AUC 0.976 (precision / recall 0.989 / 0.805 at 0.5); BIPIA email AUC 1.000, recall **0.825** at 0.5; hand-labelled calls AUC 0.993; 68 benign texts about injection gave 8 false positives at 0.5 (bench README: 7) | Ranking | Raw responses published. Policy-violation, off-task and tool-description questions unmeasured |
| deepset, LLMail, BIPIA EmailQA, held-out tool results ([jev-edge](https://github.com/kiwi0719/jev-edge), kiwi0719) | A live Jev check behind cheap rules in an OpenResty gateway | deepset AUC 0.983 with 37.3% missed; **0.996 and 5.3% missed** once the deployment is described (as jev-sec-bench above). LLMail 29.8% missed; BIPIA EmailQA **81.5% missed**; tool results 77.8% → 19.0% missed when marked untrusted | Context line wins | One run each. Misses match [[concepts/jaggedness-jev-1-13]] §6: `verified` in direction. BIPIA results conflict with [agent-chaperone](https://github.com/agent-chaperone/agent-chaperone)'s (BIPIA email recall 0.825 at 0.5 vs 81.5% missed): different question wording and BIPIA task; cite both |
| Judged audit contests USSD, Beedle, Monolith ([jevscan-evm](https://github.com/devtooligan/jevscan-evm), devtooligan) | 359 Nouls per Solidity file (2 general + 14 category + 343 checklist-derived) vs contest findings; vs GPT-5.6 high on one file, 14 checks | Findings located at file level ≥ 70%: USSD 20/22 (0 false positives), Beedle 27/42 (1), Monolith 6/7 (3); function level 16, 18, 5. One file, 14 checks: 0.7 s and 0.015¢ vs 57 s and 3.9¢ | Triage heatmap | Author says the code is vibe-coded and unread. Batch stability: [[ideas/failure-reports-confidence]] |

## How to read these

- **A deployment line moves the cut, not the ranking** (lesson: [[ideas/builder-lessons]]): jev-sec-bench and jev-edge both gained most once the assistant's purpose was in `state`; AUC barely moved.
- **Wording and task decide injection recall**: jev-edge and agent-chaperone disagree on BIPIA email; cite both.
- **Simple baselines first**: a two-rule regex beat Jev's best single phishing signal; trained regression matched Jev on spam with labels and a stable corpus.
- Patterns: [[ideas/patterns-data]] P16 screening; [[ideas/patterns-agents]] P03 safety gates; tools: [[ideas/tools-guardrails]].

## Related

- [[ideas/head-to-head-benchmarks]]: the other public sets (intent, retrieval, science, robots, reasoning); [[ideas/head-to-head]]: app-level comparisons; [[ideas/eval-boards]]: boards
- [[ideas/failure-reports]], [[ideas/failure-reports-confidence]]: where these broke; [[ideas/cost-ledger]]: cost per decision
- [[concepts/jaggedness-jev-1-13]], [[concepts/confidence]]

## Sources

Links inline; raw captures listed in the frontmatter (private repo).