Field benchmarks (labs)¶

Transparency first

These numbers come from reproducible lab harnesses, not marketing claims. Each row links to the raw JSON artifact. We distinguish autonomous discovery (generic crawl + DEPTH-16) from lab-specific confirmation probes (known challenge solutions). When a target sanitizes output, we report OBSERVED, not CONFIRMED — we do not inflate execution.

Last updated: 2026-07-24 (UTC) · Known labs: data/bench_*.json via python scripts/regen_benchmarks_docs.py · Novel targets: GEN-01b harness (scripts/bench_novel_target.py) — metrics published after a real run (GEN-01d).

Summary table (known labs)¶

Lab / gate	Metric (honest)	Mode	Gate	Reproduce	Raw JSON
Google Firing Range	Reflected 15 browser-confirmed / 40 endpoints seen · DOM 3 confirmed (ceiling 1 stable on live lab)	Autonomous discovery + DEPTH-16 dialog	PASS	`python scripts/bench_firing_range.py --measure-only`	assets/benchmarks/firing_range.json
XSS Game	6/6 levels browser-confirmed (dialog)	Lab-specific probes — not autonomous discovery	PASS	`python scripts/bench_xss_game.py --measure-only`	assets/benchmarks/xss_game.json
webscantest.com	Coverage 68% authenticated	Docker-local (`127.0.0.1`); public host returns 403 to scanners	PASS	`python scripts/bench_webscantest_auth.py --measure-only`	assets/benchmarks/webscantest_auth.json
OWASP Juice Shop (DEPTH-08)	? OBSERVED · 0 CONFIRMED	Store→render pipeline; target sanitizes stored XSS	FAIL	`python scripts/bench_admin_sinks.py`	assets/benchmarks/admin_sinks.json
Anti-FP corpus (BENCH-REAL)	0 FP · precision@confirmed 1.0 (29/29 cases)	Fixture corpus (no live targets)	PASS	`python scripts/bench_real.py --fixtures-only`	assets/benchmarks/real_report.json
Parabank (BENCH-02)	`parabank_recall` probe PASS (public + local Docker)	Targeted recall probe, not full-stream gate JSON	PASS	`python scripts/benchmark_recall.py --include-labs`	BENCH-02 probe

What each number means¶

Firing Range — autonomous XSS depth¶

Reflected 15/40: DEPTH-16 Playwright dialog on distinct reflected paths from the seed probe surface (not the whole lab catalog).
DOM 3 confirmed, ceiling 1: live toxicdom matrix; innerHtml sink is intermittent. We do not claim DOM ≥10 on the public Firing Range — that requires a dedicated DOM-rich lab (Ola 3).

XSS Game — lab-specific confirmation (not discovery)¶

6/6 counts only levels with browser dialog (CONF-MODEL-01).
Probes are preloaded per challenge (xss_game_lab.py) — this demonstrates cross-context confirmation (reflected, stored, DOM hash, attribute breakout, javascript: URI, gadget loader), not that the generic crawler discovers all six levels alone. Full pentest stream hit wall-clock timeout (timeout after 420.0s).

webscantest — authenticated coverage¶

68% from scan_coverage.coverage_pct with verified session (auth=True on post-login paths).
Measured against Docker-local Rapid7 image; scanning the public webscantest.com hostname is blocked (403).

admin_sinks (DEPTH-08) — honest OBSERVED¶

Pipeline exercised: discover sinks → store payload → re-render → browser verify.
This Juice Shop build strips/escapes stored payloads → 0 CONFIRMED, ? OBSERVED. Gate PASS means the pipeline is honest, not that we faked execution.

Anti-FP (BENCH-REAL)¶

29 negative fixtures (SPA catch-alls, JSON reflections, timing jitter, etc.) must produce zero CRITICAL/HIGH false positives.
precision_at_confirmed: 1.0 on the anti-FP corpus (no confirmed findings expected).

Parabank (BENCH-02)¶

Regression via parabank_recall (SQLi login bypass / session recall) in scripts/benchmark_labs.py.
No dedicated data/bench_parabank.json yet — PASS is from the BENCH-02 multi-lab probe harness.

Novel targets (GEN-01) — what a new app gets¶

Separate from known-lab gates

The rows above measure regression on labs we tune against (seed packs, lab-specific probes, or Docker-local harnesses). They do not prove depth on an arbitrary customer app. GEN-01 measures the surface-driven motor in clean mode — no seed pack, no VEX_RECALL_BENCHMARK, no lab profile.

Target	Status	Metric (honest)	Reproduce
OWASP NodeGoat	Published	coverage 100% · CONFIRMED+ 1 · OBSERVED 1 · recall_core 3/3 · recall_extended 8/9	See below

How we measure novel targets¶

Harness: python scripts/bench_novel_target.py --target <url> [--auth user:pass]
Clean motor: VEX_DISCOVER_API_SKIP_SEEDS=1; recall-benchmark env vars unset.
Ground-truth: data/truth/nodegoat_truth.yaml — recall split into:
recall_core — only dast_detectable: yes (honest DAST denominator)
recall_extended — yes + partial (auth/heuristic-dependent)
out_of_scope_dast — no entries listed separately (never inflates recall)
Anti-contamination: flags Meridian strings, lab_profile, recall_benchmark in meta/env.
Exit code 3 if target unreachable — no fabricated metrics.

git clone https://github.com/OWASP/NodeGoat.git /tmp/NodeGoat
cd /tmp/NodeGoat && docker compose up -d --build
python scripts/bench_novel_target.py \
  --target http://127.0.0.1:4000 \
  --auth user1:User1_123 \
  --json data/bench_novel_target.json
# Then: python scripts/regen_benchmarks_docs.py

NodeGoat — published run (2026-07-24)¶

Mode: autonomous surface-driven · VEX_DISCOVER_API_SKIP_SEEDS=1 · contamination CLEAN
Scan: 14 findings · coverage 100% · elapsed 680.2s (local Docker · Chromium OK)
Confidence: CONFIRMED+ 1 · OBSERVED 1 · HIGH+ 3
Recall core (dast_detectable: yes only): 3/3 — honest DAST denominator
Recall extended (yes + partial): 8/9 — includes two table-stakes-to-moderate generic checks labeled honestly: NG-A5 (Security Misconfiguration) via a missing-security-header finding (CWE-16), and NG-A2 (weak password policy) via a generic weak-password probe on the discovered registration form (CWE-521, OBSERVED — server accepts trivial passwords). The one remaining partial, NG-A7 (function-level access control), needs role comparison / 2-identity; interpret with care
Out of scope DAST (dast_detectable: no): 2 entries — not counted as motor failure
Caveats: playwright_crawl OK (13 pages, 6 DOM forms) · 6 injection point(s) on bus · discover_api: 0 API endpoint(s) · sources={} · recall_core 3/3 on NodeGoat truth (dast_detectable: yes) — surface-driven motor (GEN-02): SSJI eval timing, profile write XSS, open redirect · no seed pack · truth matching is keyword/CWE heuristic
Raw JSON: assets/benchmarks/novel_target.json

Finding adjudication (manual review)¶

XSS /profile (firstName): TP write-reflection — authenticated POST reflects unsanitized markup in the HTML response (admin_sinks write probe). HIGH confidence, not CONFIRMED (no browser dialog on render). Matches truth NG-A3-XSS-STORED (dast_detectable: yes).
SSJI /contributions: TP timing-confirmed — server-side eval busy-loop delay detected on contribution fields. CONFIRMED (matches truth NG-A1-SSJI).
XSS /signup (userName): TP reflejado — input echoed without encoding on validation error (NodeGoat session.js; ESAPI disabled in source). HIGH confidence (reflection detected), not CONFIRMED (no browser dialog; value= attribute context). Distinct from truth NG-A3-XSS-STORED (POST /profile). Tracked as NG-A3b-XSS-REFLECTED-SIGNUP (dast_detectable: partial).
Security misconfiguration (headers): TP — response missing hardening headers (e.g. X-Frame-Options). MEDIUM, generic (any app without these headers triggers it), tagged CWE-16. Matches truth NG-A5-MISCONFIG (dast_detectable: partial — "headers yes, internal config no"). Table-stakes finding; credited to extended recall but not equivalent to the injection-class hits.
Weak password policy (signup): TP — the discovered registration form accepts trivial industry-common passwords (123456, password, …) with no complexity block. OBSERVED, CWE-521, generic (probes any registration form found on the surface — no hardcoded path). Matches truth NG-A2-BROKEN-AUTH (dast_detectable: partial). Real product check, not a lab number; still labeled OBSERVED (acceptance detected, not full account-takeover chain).

GEN-01a — anti-FP on lab-flavored probes (internal, affects all scans)¶

Probes that historically smelled like Meridian tuning are now evidence-gated:

Password reset abuse — CONFIRMED / password_reset_forged only after isolated re-login succeeds.
JWT confusion — forged sub from session JWT / observed email, not hardcoded lab identities.

This is not a separate public gate row; it protects novel-target honesty.

Enterprise scorecard (known labs)¶

Gate	Status	Notes
G1 Meridian recall	🟡 22/23	Stretch 23/23
G2 Firing Range	🟢	15/40 refl + DOM 3 (ceiling 1)
G3 XSS Game	🟢	6/6 lab-specific probes
G4 webscantest auth	🟢	68% Docker-local
G5 Honest reporting	🟢	RPT-09/10
G6 HIGH+ evidence	🟢	85.0% on lab corpus (`scripts/bench_g6_evidence.py`)
G7 jsonplaceholder anti-FP	🟢
G8 pytest	🟢	≥2982
G9 Public benchmarks	🟢	This page
G10 Parabank BENCH-02	🟢	`parabank_recall` PASS
G11 OWASP taxonomy	🟢
G12 XSS browser-executed	🟢	DEPTH-16 + D4
G13 SAFETY-01 process	🟢	Documented in release checklist

Score: 12 / 13 green (92% — G6 measured 07-18). Yellow: G1 (Meridian recall stretch). Red: none.

What we do not claim¶

DOM XSS ≥10 on the live Firing Range — public lab ceiling is 1 stable sink(s); three dialogs were observed but only one is gated as stable.
Autonomous 6/6 on XSS Game — our 6/6 uses lab-specific solvers; autonomous metric is Firing Range reflected recall.
CONFIRMED stored XSS on sanitizing apps — Juice Shop admin_sinks correctly stays OBSERVED.
webscantest coverage on the public hostname — blocked; Docker-local only.
100% recall on every OWASP class in every customer app — labs are regression targets, not guarantees.
Novel-target depth is measured separately (GEN-01) — see NodeGoat section below; not comparable to tuned lab gates.

Regenerate monthly¶

# Re-run benches (labs must be reachable / Docker up as noted), then:
./venv/bin/python scripts/bench_firing_range.py --json data/bench_firing_range.json
./venv/bin/python scripts/bench_xss_game.py --json data/bench_xss_game.json
./venv/bin/python scripts/bench_webscantest_auth.py --json data/bench_webscantest_auth.json
./venv/bin/python scripts/bench_admin_sinks.py --json data/bench_admin_sinks.json
./venv/bin/python scripts/bench_real.py --fixtures-only --json data/bench_real_report.json
# Novel target (GEN-01b) — optional; requires Docker NodeGoat up:
./venv/bin/python scripts/bench_novel_target.py \
  --target http://127.0.0.1:4000 --auth user1:User1_123 \
  --json data/bench_novel_target.json
./venv/bin/python scripts/regen_benchmarks_docs.py
cd docs-site && mkdocs build --strict