Agents: read the raw Markdown of this page, or start at llms.txt.
~/wiki/ideas
Head-to-head benchmarks: Jev on public datasets and independent eval boards
TL;DR On public sets Jev is a strong cheap zero-shot baseline, not a frontier replacement: it trails frontier LLMs by 1-6 points and fine-tuned encoders by 7-13 on fine-grained intent (Banking77) and law (LexGLUE), ties or leads on short crisp labels, out-of-scope detection, reranking and injection ranking, and loses when raw numbers or a hand-written regex carry the signal. Its confidence ranks well; calibration varies by dataset. App-level comparisons, cascades and the cost ledger: Head-to-head: Jev against other models and methods.
Numbers as each builder published them, on jev-1.13.0 unless stated. LOST = Jev lost on the stated metric. Every set here is public and old enough to be in some model's pretraining (several authors say so); one run each. Vendor or promotional sources are labelled.
Independent eval boards
| Board (operator) | Task | Numbers as published | Jev placed | Caveat |
|---|---|---|---|---|
| Jevals.com noul board (anonymous operator; method) | PubMedQA, 300 items × 5 repeats, human labels | Decision Score (0 = guessing base rates) Jev 69.0 vs Gemini 3.8 Flash 73.0; accuracy 91.3% vs 92.5%; $0.029 vs $0.8 per 1,000; p95 653 ms vs 5.5 s. At 95% accuracy Jev can act alone on 86% vs 79% | Tied first: paired difference −2.5 to +10.4 includes 0 | Operator anonymous, site carries a job-board ad; no frontier LLM listed (budget $25). LLM probabilities are verbalized, not logprobs |
| Jevals.com choice board | Banking77, 77 intents, 300 × 5 | Jev 67.8 vs Gemini 3.8 Flash 74.1 vs GLM-5.3 66.8; accuracy 79.7% vs 84.6%; $0.043 vs $1.4 per 1,000; 41.5% of Jev answers at confidence 1.00, ECE 9.8 vs 3.2 points; 10.3% of Jev picks change when options are reordered | LOST to Gemini (significant); tied 2nd with GLM-5.3 | Options shuffled identically for every system; 2.7% of Jev picks change on an identical rerun |
| Jevals.com score board | HelpSteer2 helpfulness, 5 levels | Jev 9.2, GLM-5.3 7.8, Gemini 4.6: none beats guessing the label base rates; Jev accuracy 41.3%, ECE 19.7 | Nobody wins | No model reaches 95% accuracy at any gate; keep a person in the loop |
| JevBench, 534 frozen decisions (fstandhartinger) | 48 ranked systems | Jev 1.13.0 #1 at 74.4; SemIf 73.1; djev 73.0 | Composite | Mixes accuracy, calibration, speed, cost; hard items written by Opus 5 and GPT-5.6 Sol; self-hosted latency ×2 + 0.15 s by assumption |
Reading Jevals: the boards list Jev first on both noul and choice because they order by "wins" (top-2 finishes across score, accuracy, calibration gap, price and speed), not by Decision Score. The capture says PubMedQA is a statistical tie with Gemini 3.8 Flash (Jev lower on the point estimate) and Banking77 a real loss, so "behind Gemini Flash on both" overstates PubMedQA. Its 67.8 vs 74.1 on Banking77 is a Decision Score and only coincides with TypeSafe's workflow-eval 67.8% vs 74.1% (Workflow evals: how TypeSafe measures Jev); different measures. Jevals and ASSAY-001 both report Jev probabilities rounded to 2 decimals (sums of 0.99): unverified, not in HTTP API: POST /v1/systemone and GET /v1/models. Jevals is not openlayer-ai/jevals.
Intent and text classification
| Dataset (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| Banking77 full test, 3,080 (OpenRouter blog, Kenny Rogers; vendor: sells both models) | Jev vs Claude Opus 5, reasoning off, strict schema | Accuracy 81.0% vs 84.4% (paired gap 2.3-4.4 pts); p50 175 vs 2,266 ms; $0.11 vs $2.42 per 1,000 (Opus prompt cached). Opus led on 35 intents, Jev 15, tie 27 | Latency, cost; LOST accuracy | Criteria from label names only; both 0/40 on get_physical_card. Says confidence is not a calibrated probability but ranks well (≥ 0.99: 58% of items, 96.3% right): verified, confidence is a statistic of the distribution, not a probability (Confidence vs probability). Cascade: Head-to-head: Jev against other models and methods |
| Banking77: 8 confusable intents (160) and all 77 (231) (AY Automate, Adel Dahani; agency blog with sales CTAs, no client use of Jev) | Jev vs GPT-5.4 nano, Gemini 3.5 Flash-Lite, Haiku 4.5, GPT-5.6 Terra, via OpenRouter | 8-way 83.8 / 90.0 / 77.5 / 78.8 / 89.4%; 77-way 78.8 / 78.4 / 78.4 / 76.2 / 84.0%. Median 0.33 s vs 0.67-1.17 s. Only Terra reliably beat Jev on 77-way (p = 0.029); nano and Terra beat it on 8-way | Level with small LLMs; LOST to Terra by ~5 pts | Labels only, no descriptions; intervals 6-13 pts wide. Five confident 8-way errors were one overlapping label pair (Terra made the same call). Jev billed more input tokens than Terra for the same prompt (360 vs 153). Its "access by waitlist" line is dated (console.typesafe.ai (console + playground)) |
Banking77 (3,080) and CLINC150 plus (5,496) calibration (ASSAY-001, write-up, JourdanLabs) |
Pre-registered, hashed protocol; blind re-score | Banking77 ECE 0.0936, overconfident in every bin below 0.9 (says 0.86, right 67%); CLINC150 ECE 0.0204, calibrated; accuracy 0.7977 / 0.8812; 0 type errors in 8,576 responses | Calibration holds on CLINC150, fails on Banking77 | Per-task exception to the docs' calibration claim: unverified in general, measured here. Failure reports: where Jev broke, lost, or was the wrong tool |
| 4 public sets × 300 (Aman Kumar) | Jev / gpt-5.4-mini / gpt-5.6-luna | Enron 98.7 / 97.7 / 98.0; SST-2 95.7 / 92.7 / 93.0; AG News 91.3 / 88.3 / 89.7; Banking77 76.0 / 78.7 / 81.7. Median 0.8-0.9 s vs 1.4-5.0 s | 3 of 4 sets; LOST Banking77 | At 100 in flight a few calls per thousand took 10-35 s |
| 5 labelled tasks × 300 (4esv) | Jev / GPT-5.6 Terra / open-jev 435M / Kev-0.8B / Laya 421M | CLINC-150 0.897 / — / 0.610 / 0.643 / 0.497. Banking77 0.780 / 0.847 / 0.873 / 0.770 / 0.370. SST-5 0.570 / 0.593 / 0.560 | CLINC: Jev. LOST Banking77 and SST-5 | open-jev and Kev trained on Banking77 and SST-5. Only Jev's latency stayed flat from 2 to 151 options (0.17 → 0.20 s). open-jev was best calibrated on 4 of 5 |
| LexGLUE, all 7 tasks, 23,607 test examples (chepyle/jev-test, via OpenRouter) | Jev vs GPT-5.6 Luna (JSON schema) vs fine-tuned BERT-base | Mean micro-F1 69.9 vs 71.3 vs 77.4; $4.02 vs $16.45 vs $21.58 GPU. Per-label thresholds tuned on validation: 74.2. Outside law: Banking77 80.6 vs BERT 93.7; CLINC150 in-scope 89.0 vs 96.7 but out-of-scope recall 88.1 vs 59.2 | Cost, out-of-scope; LOST to BERT everywhere else, near Luna | Zero-shot vs supervised; Luna led ECtHR A/B, EUR-LEX, UNFAIR-ToS, Jev led SCOTUS and LEDGAR, CaseHOLD tied. Long documents cut to head+tail 48,000 chars |
| 8 datasets, 3 seeds (QuicqDev) | vs 11 trained classical pipelines | IMDb 96.3% vs 88.4% balanced accuracy; classical models led on all 4 tabular sets | Text: Jev. LOST tabular | Threshold-adjusted rows use labelled data |
| JNLI (JGLUE), 2,434 test (agentjournal, ikkun) | One direct Choice vs 14 Jev-scored dimensions + fitted weights vs n-grams | 0.837 vs 0.908 vs ≤ 0.553 (majority class 0.553); contradiction 0.654 vs 0.921 | Dimensions + weights | Author's hand-written dimensions; 55% neutral skew. Bookkeeping part: Head-to-head: Jev against other models and methods |
| AG News, Banking77, 6-label emotion (jev-benchmarks, AbdelStark) | vs local GLiNER2.5 | see Failure reports: where Jev broke, lost, or was the wrong tool (confidence misread) |
Security: phishing, spam, injection, vulnerable code
| Dataset (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| PhishNChips v5.2, 2,000 emails (anisselbd/jev-phishing-bench; the study Aman Kumar relays) | Jev verdict Choice vs Claude Haiku 4.5; then 5 signal Nouls + regression; controls added after review | Verdict 62.6% vs 81.3% (McNemar p < 0.0001); ECE 0.154 vs 0.097; p50 239 vs 687 ms; $0.038 vs $0.462 per 1,000. On a held-out half: Jev's five signals + regression 95.0%; Haiku asked the same five questions 93.2% (p = 0.063, tie; Haiku AUROC higher); a two-rule regex 91.8% beats Jev's best single signal 89.4% (p = 0.003) | LOST verdict; signals tie Haiku at ~27× less cost | Synthetic bodies, labels from URL feeds, signals written after reading the dataset's URL taxonomy; the regex alone scores 91.6% on all 2,000. No licence (describe only) |
| Ham / spam / phishing, 9,886 messages (bitnovus/jev-spam-eval) | Jev zero-shot Choice vs TF-IDF logistic regression | Main test (5,733): 98.64% vs 98.87%. Adding link targets, Reply-To and attachment metadata, same question: 93.62% → 97.98%, phishing recall 85.71% → 98.43%. 853 phishing mails from 2024-25: 95.31% vs 75.26% recall. 50/50 average: 99.30% | Tie on main; recall on recent phishing | Wording informed by earlier labelled errors; regression kept the lead on fresh mail (95.08% vs 93.83% without near-copies); recent set has no legitimate mail |
| Spam, 19,500 emails (relayed by Kumar) | vs logistic regression trained on 15,000 | 98.3% zero-shot, level | Tie with no training | Original not captured. bitnovus's earlier binary run (18,514 unique messages, 98.33% vs 98.39%) may be this study (inferred); kept apart |
| deepset/prompt-injections, all 662 (jev-sec-bench, Gaurav Gosain) | Same Noul with vs without a line saying what the assistant is for | At a plain 0.50 cut: 96.5% vs 89.7%, recall 95.1% vs 74.9%, ROC-AUC 0.9927 vs 0.9846, ECE 0.0588 vs 0.0928 | Context line wins | Under-confident at the top (said 0.85, right 100%). Vulnerable-code pairs: vulnerable half scored above its secure twin in 178/200, absolute accuracy 71.5%; ≥ 38% of false positives are corpus label errors per the author (unverified) |
| deepset/prompt-injections, 400 of 662 (AY Automate, agency, as above) | vs the four LLMs above | Jev AUROC 0.990 (best of five); at 0.50 recall 0.69, precision 1.00; at 0.10 recall 0.96, precision 0.92 | Ranking; beat nano, tied the rest on accuracy | Cutoffs picked after seeing results; dataset mixes English and German, labels partly persona prompts |
| Hard-benign texts that mention attacks (339) and hackaprompt (1,383) (agentjournal) | One direct Choice vs 12 Jev dimensions + weights vs word-bigram NB | Hard benigns false-positive rate 1.5% vs 37.2% vs 43.4%; hackaprompt 0.665 vs 0.977 vs 0.994; external set 0.956 vs 0.840 vs 0.381 | Direct question on benign text; dimensions on attack strings | Author's own dimension design; four repair attempts failed |
Retrieval, search and screening
| Dataset (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| Rerank BM25 top 30, 8 datasets, 1,617 queries (anessbelbati) | vs Cohere Rerank 4 Pro, zerank-2, DeepSeek | nDCG@10 0.692 / 0.691 / 0.682 / 0.682; BM25 0.486. 422 vs 844 ms; $0.45 vs $2.51 per 1,000 queries | Tie; latency. Per-query weighting: LOST (0.738 vs 0.756) | Reversing passage order changed Jev's top pick on 24.7% of queries |
| Reranking, NevIR negation (anessbelbati) | vs Open-Jev 9B, Laya | nDCG: Jev yes/no per pair beats Open-Jev 9B by 7.0 points; Laya ≈ BM25. Negation pairs: Open-Jev 9B 77% vs Jev 71%, Laya 36% | nDCG: Jev. LOST negation to 9B | Open-Jev 9B: 19.1 s per query |
| BEIR NFCorpus (323) and SciFact (300) (llama-index-jev, WiktorB2004) | MiniLM top-10, Jev Score rerank to top 5, vs MiniLM and rank-bm25 |
nDCG@5 NFCorpus 0.340 → 0.396 (+0.056, CI 0.042-0.072); SciFact 0.629 → 0.715; BGE-small 0.375 → 0.415. ~$0.0003 per query (OpenRouter usage.cost) |
Beats its first stage | Not a BEIR leaderboard run (no Pyserini, no Cohere column); rerank fails open |
| SWE-bench Lite file localisation, frozen 202-issue test split (siftr, Bentlybro; method) | Jev outlines search vs BM25 vs grep | Right file in top 5: 82% vs 52% vs 22%; pick test file among ~550: 81% vs BM25 38%; read kept 92% of edited lines while cutting 59%. ~2 s, 1-2¢ per search |
All; LOST on scikit-learn (71% vs 79%) | Log filter lost to grep (both kept 100% of alert lines; grep at 23% of the log vs 46%). Snippets go to OpenRouter |
| CLEF TAR 2019, 12 Cochrane reviews (Paper Radar, Eliot5566) | Jev criteria Nouls vs reviewers' title/abstract decisions | 4 held-out reviews: 19,447 records, 96.9% recall, 78.0% pooled work saved, $0.60. 8 development reviews: 38-48% pooled (0.1% to 82% per review) | Held-out: yes; dev: weak | Thresholds fitted on the same judgments (optimistic); pre-registration predicted 30-70% and was wrong. Its "waitlist" line is stale. Failure reports: where Jev broke, lost, or was the wrong tool |
| Loghub HDFS and BGL samples (Jev Logs, reachjalil) | Jev line triage vs block labels | Default routing kept 0.84% of a 30%-anomalous HDFS sample (5 of 750 labelled "anomalies", all successful block verifications); BGL alerts caught 100% by the local FATAL rule, not Jev | LOST | Block-level labels vs line-level judgment. Synthetic pager win: Head-to-head: Jev against other models and methods |
Structured and scientific data
| Dataset (builder) | Compared | Numbers as published | Won | Caveat |
|---|---|---|---|---|
| NASA Kepler objects of interest, 8,054, 3 classes (gist, ipaulsmith) | Jev Choice vs a fixed 3-rule baseline vs always FALSE POSITIVE | Raw measurements as state: 54.2%, below rules 64.4%. Code-bucketed named fields + structured criteria: 72.5% (CI 71.5-73.5), CANDIDATE recall 45.0%. 322 ms median, ~$0.34 |
LOST raw; won bucketed | Round 2 designed after seeing round 1, no held-out prompt set; no per-object results published. Matches Jev 1.13 jaggedness: known failure modes on numbers |
| Who&When Pro agent-failure attribution, 6,257 text traces (TokenTrim) | Jev (3 Choices per trace) vs the paper's GPT-5.4, Claude Sonnet 4.6, GLM-5, Qwen3.5-122B | Who / when / what: Jev 73.4 / 76.4 / 23.7 vs GPT-5.4 55.7 / 72.3 / 15.3; next best "what" 22.2 (GLM-5); ~$1.28 total | "What" (the only like-for-like axis) | Jev picks from listed agents and steps while the LLMs free-generate; baselines are the paper's, not re-run; failures injected, not natural |
How to read these
- Public ≠ unseen. Banking77 dates from 2020; several authors warn of pretraining overlap for Jev and the LLMs alike.
- Banking77 is Jev's recurring weak set: 76.0-81.0% across seven builders, while the best comparator in each run scored 81.7-93.7%, with overconfidence (ASSAY, Jevals). Out-of-scope and short binary labels are its strong side.
- Structure beats raw input: bucketed Kepler fields, a deployment-context line (injection), link metadata (spam), named lines. Numbers still belong in code (Jev 1.13 jaggedness: known failure modes).
- Check simple baselines: a regex (phishing), grep (log filter), rules (Kepler raw) and n-grams (hackaprompt; bookkeeping on Head-to-head: Jev against other models and methods) matched or beat Jev somewhere.
- Patterns: Patterns: judging, search, documents, real-time and markets P16 screening, P17 rerank, P18 corpus labels; Patterns: gates, simulation, personas and other shapes (P38+) P42 tuning against labels.
Related
- Head-to-head: Jev against other models and methods: app-level comparisons, cascades, cost ledger; Failure reports: where Jev broke, lost, or was the wrong tool: where these broke
- Measurements, access routes and open replicas: latency, billing and access numbers; Workflow evals: how TypeSafe measures Jev: TypeSafe's own board
- Confidence vs probability, Testing and evaluating a Jev workflow: run your own comparison
Sources
Links inline; raw captures listed in the frontmatter (private repo).