Security Testing (ST) - Software Assessment

Assessment questionnaire for measuring maturity. Answer each question honestly based on current, implemented practices.

v3.0 framing: The canonical source-of-truth for Security Testing (ST) in the Software domain is ../practices/ST-Software-OnePager.md. Outcome metrics, activities, and success criteria below are verbatim from that one-pager. Canonical subject and through-lines: ../HAIAMM-v3.0-Framing.md.


Security Testing (ST) - Software Domain

HAIAMM Assessment Questionnaire v3.0

Practice: Security Testing (ST) Domain: Software Purpose: Assess organizational maturity in running adversarial test batteries and regression corpora against AI/HAI software artifacts, from foundational per-archetype batteries in CI through continuous automated adversarial testing Scoring Model: Evidence + Outcome Metrics (see Scoring Methodology below)


Instructions

  • Answer each question honestly based on current, implemented practices (not plans or aspirations)
  • Each question has two components: Evidence (what you did) and Outcome Metrics (how well it worked)
  • Scoring uses 4 tiers: Fully Mature (1.0), Implemented (0.67), Partial (0.33), Not Implemented (0.0)
  • Answer progressively - Complete all Level 1 questions before Level 2
  • Level progression - Achieve ALL questions at lower level before advancing
  • Baseline first - Record current metric values before setting targets

Scoring Methodology

Score Label Criteria
1.0 Fully Mature Evidence complete + ≥3 outcome metrics meet targets
0.67 Implemented Evidence complete + 2 outcome metrics meet targets
0.33 Partial Evidence partially complete + <2 outcome metrics meet targets
0.0 Not Implemented No evidence of the activity

Level Score = average of the 3 question scores for that level Overall ST-Software Score = weighted average: L1 × 0.5 + L2 × 0.3 + L3 × 0.2


Maturity Level 1

Objective: Establish a foundational per-archetype test battery and regression corpora that run in CI on every PR, and verify that every AI/HAI software artifact reaches production with a passed go-live battery on record


Question 1: Per-Archetype Test Battery

Q1.1: Is a per-archetype foundational test battery published for all seven AI/HAI software archetypes (LLM-integrated app, autonomous agent, RAG pipeline, fine-tuning/training workload, eval/red-team harness, model-serving service, classical ML model), with each test class tied to a TA-Software archetype threat (HAI TTP + ATLAS tactic ID) and an SR-Software requirement, defined inputs/outputs/pass-fail criteria, and an evidence artifact?

Evidence Required: - [ ] Test battery document published per archetype; covers per-archetype probes including prompt-injection corpus (ATLAS TA0001/TA0003), training-data leakage canary (ATLAS TA0013), tool-scope boundary test (ATLAS TA0004), agent goal-hijack probe (ATLAS TA0001/TA0003), kill-switch test, HITL gate test, RAG injection test, eval isolation test, model-serving rollback test, and classical ML drift test as applicable per archetype - [ ] Each test class records: inputs, expected output, pass/fail criteria, evidence artifact (log snippet, CI run link, or trace ID) - [ ] Each test class is mapped to a TA-Software threat (HAI TTP + ATLAS tactic ID) and an SR-Software requirement - [ ] Battery linked from SM inventory record and DR/IR artifacts for each AI/HAI software artifact - [ ] Named battery owner per archetype documented; quarterly re-run scheduled in advance - [ ] 100% of AI/HAI software artifacts reaching production in the last 90 days have a passed go-live battery on record

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | % of AI/HAI software artifacts reaching production with a passed go-live battery on record | % | % | ≥90% within 12 months; 100% for Critical/High-tier | ☐ | | | % archetype threat library entries covered by at least one test or corpus entry | % | % | ≥80% by end of year 1 | ☐ | | | % test failures routed to IM within 1 business day | % | % | 100% | ☐ | | | CI automation coverage of battery items (% running without human intervention) | % | % | ≥60% | ☐ | |

Metric Collection Guidance: - Go-live battery coverage: Query SM inventory registry for artifacts promoted to production in the last 90 days; cross-reference with the test-run registry. Formula: artifacts_with_passed_battery / artifacts_reaching_production × 100 - Threat coverage: For each archetype's TA-Software library entry, check whether at least one test class or corpus entry maps to it. Formula: covered_threats / total_archetype_threats × 100. Source: TA library × test metadata mapping - IM routing rate: For test failures generated in the last quarter, check IM ticket creation timestamps. Formula: failures_with_IM_ticket_within_1BD / total_failures × 100. Source: CI telemetry × IM system - CI automation: Count battery items with a CI trigger vs. total battery items. Source: battery registry + CI configuration

Answer: - ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets) - ☐ Implemented (Evidence complete + 2 metrics meet targets) - ☐ Partial (Evidence partially complete + <2 metrics meet targets) - ☐ Not Implemented (No evidence of per-archetype test battery)

Evidence Location: __ Validation Date: __ Notes: ___


Question 2: Regression Corpora in CI

Q2.1: Are four regression corpora (jailbreak, prompt-injection, agent goal-hijack, tool-misuse) versioned in source control, running in CI on every PR for Critical/High-tier artifacts, with a named corpus owner, a monthly refresh cadence from internal and external sources, and a CI token-spend budget cap?

Evidence Required: - [ ] Four regression corpora published in source control: jailbreak corpus (30–100 entries covering role-override, persona-switch, authority-claim, encoding-bypass), prompt-injection corpus (30–100 direct and indirect entries), agent goal-hijack corpus (20–60 multi-turn sequences), and tool-misuse corpus (20–60 argument-smuggling payloads) - [ ] Each corpus entry includes: input, expected safe output pattern, threat tag (HAI TTP + ATLAS tactic ID), OWASP reference, source, date added - [ ] CI wiring confirmed: corpus runs on every PR for Critical/High-tier artifacts; failure is a blocking check for Critical/High-tier - [ ] Named corpus owner per corpus; corpus changes go through PR review - [ ] Monthly refresh cadence evidenced: change-log showing updates from internal observations (IR findings, IM incidents), external sources (OWASP LLM Top 10, HackAPrompt, ATLAS examples), and jailbreak research repositories - [ ] CI token-spend budget cap configured and enforced; evidence of cap limit in CI configuration - [ ] New PR for AI/HAI software that changes archetype triggers a corpus completeness check

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | Regression corpora published (jailbreak, prompt-injection, AGH, tool-misuse) | /4 | /4 | 4/4 | ☐ | | | % PR merges for Critical/High-tier artifacts that ran the regression corpus and passed | % | % | ≥95% | ☐ | | | Corpus refresh cadence, months since last update per corpus | ___ | ___ | ≤1 month | ☐ | | | % archetype threat library entries covered by at least one corpus entry | % | % | ≥80% by end of year 1 | ☐ | |

Metric Collection Guidance: - Corpus publication: Count corpora present in source control with versioned entries and a named owner. Source: corpus registry in VCS - PR corpus run rate: Pull CI run history for Critical/High-tier artifact PRs; cross-reference with corpus run results. Formula: PRs_with_corpus_run_and_pass / total_Critical_High_PRs × 100. Source: CI telemetry - Refresh cadence: Inspect git log for each corpus; compute days since last commit adding new entries. Source: VCS change-log - Threat coverage: TA library entries mapped to corpus entries vs. total archetype library entries. Source: TA library × corpus metadata

Answer: - ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets) - ☐ Implemented (Evidence complete + 2 metrics meet targets) - ☐ Partial (Evidence partially complete + <2 metrics meet targets) - ☐ Not Implemented (No regression corpora wired into CI)

Evidence Location: __ Validation Date: __ Notes: ___


Question 3: Go-Live Battery and IM Wire

Q3.1: Is the go-live battery operated with defined re-run triggers (pre-production, post-model-update, post-incident, quarterly), and are all test failures routed to IM-Software within 1 business day with a severity tag and named owner?

Evidence Required: - [ ] Go-live battery process document defines re-run triggers: pre-production (blocks Sanctioned status), post-model-update within 14 days (Critical-tier: 7 days), post-incident before incident closure, and quarterly for all active artifacts - [ ] Go-live test records linked from SM inventory and PC intake artifacts for artifacts promoted in the last 90 days - [ ] Test failure to IM routing confirmed: sample of recent failures shows IM tickets created within 1 business day with severity tag (Blocker/Critical/High/Medium/Low), named owner, and test battery reference - [ ] Severity rubric published and applied consistently to test failure triage - [ ] Battery re-run cadence evidenced: quarterly re-run records on file for active artifacts with named battery owner - [ ] Post-model-update re-run triggered and completed within SLA for any model-version changes in the last 6 months

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | % test failures routed to IM within 1 business day | % | % | 100% | ☐ | | | % post-model-update re-runs completed within declared SLA (7/14 days by tier) | % | % | ≥90% | ☐ | | | % active artifacts with a quarterly battery re-run completed in last 90 days | % | % | ≥90% | ☐ | | | % of AI/HAI software artifacts reaching production with a passed go-live battery on record | % | % | 100% for Critical/High-tier | ☐ | |

Metric Collection Guidance: - IM routing rate: Compare test failure log timestamps to IM ticket creation timestamps. Source: CI failure log × IM system timestamp query - Post-model-update SLA: For model-version changes in last 6 months, check re-run date vs. change date by tier. Source: model registry changelog × test-run registry - Quarterly re-run rate: Count active artifacts with a battery run record dated within the last 90 days. Source: test-run registry × SM inventory active artifact list - Go-live coverage: SM inventory × test-run registry cross-reference for artifacts reaching production in last 90 days

Answer: - ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets) - ☐ Implemented (Evidence complete + 2 metrics meet targets) - ☐ Partial (Evidence partially complete + <2 metrics meet targets) - ☐ Not Implemented (No go-live battery or IM wiring in place)

Evidence Location: __ Validation Date: __ Notes: ___


Maturity Level 2

Objective: Calibrate test depth per risk tier using the SM L2 tier-treatment matrix, run scheduled red-team exercises per tier using TA L2 deep threat models, and test cross-archetype compositions for Critical-tier artifacts


Question 4: Tier-Calibrated Battery and Corpus Depth

Q4.1: Is per-tier corpus calibration enforced in CI (Critical-tier: all 4 corpora on every PR + daily output-integrity regression; High-tier: all 4 corpora on merge; Low-tier: jailbreak corpus on merge), and does each Critical-tier artifact have a separately tuned corpus derived from its TA L2 per-artifact deep threat model?

Evidence Required: - [ ] Per-tier test treatment published and aligned to SM L2 tier-treatment matrix: Critical (full battery + all 4 corpora every PR + daily output-integrity + logging-completeness quarterly), High (full battery + all 4 corpora on merge + weekly output-integrity), Medium (subset battery + jailbreak + prompt-injection on merge), Low (spot-check + jailbreak on merge) - [ ] CI pipeline configuration confirms tier-differentiated corpus assignments, not the same corpus applied to all tiers - [ ] Critical-tier corpus entries are tuned to the artifact's specific tool set, retrieval sources, and data classes from the TA L2 per-artifact threat model - [ ] Daily output-integrity regression run logs on file for Critical-tier artifacts; drift alert wired to IM - [ ] Logging-completeness verified quarterly for Critical-tier; semi-annually for High-tier; evidence on file - [ ] Per-tier SLA adherence tracked and reported to program sponsor

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | % Critical-tier artifacts with all 4 corpora running on every PR | % | % | 100% | ☐ | | | Per-tier SLA adherence for testing activities (go-live battery, model-update re-run, red-team cadence) | % | % | ≥90% per tier | ☐ | | | % Critical-tier artifacts with a separately tuned corpus from TA L2 threat model | % | % | 100% | ☐ | | | % Critical-tier artifacts with daily output-integrity regression active | % | % | 100% | ☐ | |

Metric Collection Guidance: - Corpus per-PR rate: CI telemetry for Critical-tier PRs; verify all 4 corpus runs appear in each PR's check suite. Source: CI telemetry - SLA adherence: Program telemetry across go-live battery turnaround, model-update re-run dates, and red-team calendar adherence. Formula: activities_completed_within_SLA / total_activities × 100 - Tuned corpus coverage: Count Critical-tier artifacts with a corpus file containing artifact-specific entries vs. total Critical-tier artifacts. Source: corpus registry - Daily regression: Check CI schedule configuration for output-integrity job; verify last-run timestamp per Critical-tier artifact is within 24 hours. Source: CI job run history

Answer: - ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets) - ☐ Implemented (Evidence complete + 2 metrics meet targets) - ☐ Partial (Evidence partially complete + <2 metrics meet targets) - ☐ Not Implemented (No tier-calibrated corpus differentiation in place)

Evidence Location: __ Validation Date: __ Notes: ___


Question 5: Scheduled Red-Team Exercises

Q5.1: Are 100% of Critical-tier AI/HAI software artifacts red-teamed at least quarterly, and 100% of High-tier semi-annually, with scope derived from TA L2 per-artifact deep threat models covering prompt-injection chains, indirect-prompt-injection via RAG, agent tool abuse, multi-turn agent goal-hijack probes, data-egress canaries, and cross-tenant isolation, with findings routed to IM and remediation tracked?

Evidence Required: - [ ] Red-team schedule on calendar covering all Critical-tier (quarterly) and High-tier (semi-annual) artifacts; no Critical-tier artifact skipped in the last 12 months - [ ] Red-team scope documented per exercise: written rules of engagement, test plan reviewed with artifact owner, scope derived from TA L2 per-artifact deep threat model (not archetype snapshot) - [ ] ATLAS tactic IDs referenced in scope documentation: TA0001 Reconnaissance, TA0003 Initial Access, TA0004 ML Model Access, TA0012 ML Attack Staging, TA0013 Exfiltration - [ ] Red-team execution log and structured findings report on file: severity, root cause, ATLAS tactic ID, SR requirement traced, remediation pairing - [ ] Findings routed to IM-Software with severity tag and named artifact owner as assignee; remediation tracked - [ ] Scheduled red-team exercises confirmed for L2 cadence: quarterly for Critical, semi-annual for High

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | % Critical-tier artifacts red-teamed in last 90 days | % | % | 100% | ☐ | | | % High-tier artifacts red-teamed in last 180 days | % | % | 100% | ☐ | | | % red-team findings (Critical/High severity) converted to corpus entries within 30 days | % | % | ≥90% | ☐ | | | Per-tier SLA adherence for red-team cadence | % | % | ≥90% | ☐ | |

Metric Collection Guidance: - Critical red-team rate: Count Critical-tier artifacts with a red-team report dated within the last 90 days vs. total Critical-tier artifacts. Source: ST records - High red-team rate: Same logic for 180 days. Source: ST records - Finding to corpus conversion: For Critical/High severity findings, count those with a corresponding corpus entry committed within 30 days. Formula: findings_with_corpus_entry / total_Critical_High_findings × 100. Source: finding → corpus pipeline telemetry - SLA adherence: Compare scheduled red-team dates to actual execution dates across all tiers. Source: program calendar × ST records

Answer: - ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets) - ☐ Implemented (Evidence complete + 2 metrics meet targets) - ☐ Partial (Evidence partially complete + <2 metrics meet targets) - ☐ Not Implemented (No scheduled red-team function in place)

Evidence Location: __ Validation Date: __ Notes: ___


Question 6: Cross-Archetype Composition Tests

Q6.1: Are cross-archetype composition tests (agent + RAG, fine-tune + model-serving, multi-agent orchestration) documented and executed for all Critical-tier composite artifacts, with each Critical/High-severity red-team finding producing a corpus entry within 30 days?

Evidence Required: - [ ] Cross-archetype composition test plan published for each Critical-tier artifact with composite archetypes; reviewed by named architect - [ ] Agent + RAG composition test: indirect-injection via RAG path into agent goal-hijack chain exercised and documented - [ ] Fine-tune + model-serving composition test: membership-inference probe and canary-extraction probe against the served fine-tuned model documented - [ ] Multi-agent orchestration test: sub-agent goal-hijack via orchestrator response; scope-inheritance test (sub-agent cannot exceed parent scope) documented - [ ] Finding to TA library-gap pipeline: Critical gaps closed within 30 days; High within 60 days; tracked tickets on file - [ ] Regression corpus growth rate evidenced: Critical-tier corpora show ≥1 new entry per month from red-team or incident findings

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | % Critical-tier composite artifacts with documented cross-archetype composition tests | % | % | 100% | ☐ | | | Regression corpus growth rate, Critical-tier corpora (new entries per month) | ___ | ___ | ≥1 per month | ☐ | | | % red-team findings (Critical/High severity) converted to corpus entries within 30 days | % | % | ≥90% | ☐ | | | % TA library gaps (Critical tier) closed within 30 days | % | % | 100% | ☐ | |

Metric Collection Guidance: - Composition test coverage: Count Critical-tier artifacts with a composite archetype that have a documented composition test plan + execution record vs. total composite Critical-tier artifacts. Source: ST records - Corpus growth: Git log of Critical-tier corpus files; count commits adding new entries per calendar month. Source: VCS change-log per corpus file - Finding to corpus conversion: Finding log × corpus commit-log cross-reference. Source: IM findings × corpus change-log - TA gap closure: TA library-gap ticket backlog filtered to Critical tier; measure open-to-close duration. Source: issue tracker

Answer: - ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets) - ☐ Implemented (Evidence complete + 2 metrics meet targets) - ☐ Partial (Evidence partially complete + <2 metrics meet targets) - ☐ Not Implemented (No cross-archetype composition tests in place)

Evidence Location: __ Validation Date: __ Notes: ___


Maturity Level 3

Objective: Operate continuous automated adversarial testing for Critical-tier artifacts, publish regression corpora and findings as open artifacts, and contribute discovered TTPs to MITRE ATLAS, AVID, and OWASP LLM/Agentic Top 10


Question 7: Continuous Automated Adversarial Testing Harness

Q7.1: Are ≥80% of Critical-tier AI/HAI software artifacts under continuous automated adversarial testing with daily probe execution, using prompt-injection generators, indirect-injection seeders, tool-misuse generators, and output-integrity monitors, with novel TTPs triaged into the TA-Software library within 14 days and high-severity automated findings routed to IM within 24 hours?

Evidence Required: - [ ] Automated adversarial testing harness deployed and producing daily probe results against Critical-tier artifacts: prompt-injection generator (mutation + template + jailbreak-ladder generation), indirect-injection seeder (poisoned-document payloads seeded into retrieval path), tool-misuse generator (argument-smuggling variants per declared tool type), output-integrity monitor (daily golden test set run against production endpoints) - [ ] Continuous adversarial testing telemetry: harness health dashboard showing % Critical-tier artifacts with a fresh probe result within the last 24 hours; on-call paged when a feed goes stale >24 hours - [ ] Finding triage process: named ST owner reviewing automated findings at least weekly; novel TTP patterns forwarded to TA L3 auto-proposal pipeline within 14 days - [ ] High-severity automated findings route to IM within 24 hours; evidence: IM ticket timestamps vs. harness alert timestamps - [ ] ATLAS tactic walk records: harness exercises TA0001, TA0003, TA0004, TA0012, TA0013 technique coverage daily - [ ] Output-integrity drift alert wired to IM; P1 alert triggers full regression corpus re-run

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | % Critical-tier artifacts under continuous automated adversarial testing (daily probe execution) | % | % | ≥80% | ☐ | | | New-TTP ingestion lead time (automated finding to TA library entry) | ___ days | ___ days | ≤14 days | ☐ | | | % high-severity automated findings routed to IM within 24 hours | % | % | 100% | ☐ | | | Continuous harness health (% Critical-tier artifacts with fresh probe result in last 24 hours) | % | % | ≥95% | ☐ | |

Metric Collection Guidance: - Continuous coverage: Harness telemetry, count Critical-tier artifacts with a probe result in the last 24 hours vs. total Critical-tier. Source: ST harness telemetry - TTP ingestion lead time: For novel patterns identified by the harness, measure time from harness alert to TA library entry. Source: harness to TA pipeline telemetry - IM routing within 24h: Compare harness alert timestamp to IM ticket creation timestamp for high-severity findings. Source: harness alert log × IM system - Harness health: Monitoring dashboard showing last-probe-time per Critical-tier artifact; stale feeds trigger on-call page

Answer: - ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets) - ☐ Implemented (Evidence complete + 2 metrics meet targets) - ☐ Partial (Evidence partially complete + <2 metrics meet targets) - ☐ Not Implemented (No continuous automated adversarial testing harness in place)

Evidence Location: __ Validation Date: __ Notes: ___


Question 8: Industry Contributions

Q8.1: Has the program contributed ≥4 anonymized, legally-vetted findings per year to MITRE ATLAS, AVID, or OWASP LLM/Agentic Top 10, with at least one accepted as a new or refined technique, and are ≥4 open regression corpora published under a permissive license and maintained upstream?

Evidence Required: - [ ] Contribution log on file: ≥4 submissions per year to MITRE ATLAS (novel technique observations with ATLAS tactic IDs), AVID (structured vulnerability disclosures), or OWASP LLM/Agentic Top 10 (real-world telemetry evidence during revision cycles) - [ ] At least one submission accepted by ATLAS, AVID, or OWASP as a new or refined technique; evidence: acknowledgment or technique ID assignment - [ ] Legal-vetting record for each contribution: org identity scrubbed; coordinated disclosure completed for any third-party component involvement - [ ] ≥4 open regression corpora published under a permissive license (jailbreak, prompt-injection, AGH, tool-misuse); published versions scrubbed of org-specific tool names, data classes, and artifact identifiers - [ ] Published corpora maintained upstream with documented update cadence; internal corpora are a superset of published versions - [ ] Industry-contribution pipeline: at least one anonymized finding in-preparation, in-legal-review, or submitted at all times

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | Industry contributions per year (MITRE ATLAS / AVID / OWASP) | ___ | ___ | ≥4 | ☐ | | | Open regression corpora published and maintained upstream | /4 | /4 | ≥4 corpora published | ☐ | | | Contributions accepted as new or refined techniques | ___ | ___ | ≥1 per year | ☐ | | | Published corpus recency, months since last upstream update | ___ | ___ | ≤1 month | ☐ | |

Metric Collection Guidance: - Contribution count: Contribution log filtered to the last 12 months; count submissions with submission confirmation. Source: contribution log - Open corpus publication: Confirm ≥4 corpora exist in the designated public repository under a permissive license. Source: external repository - Acceptance rate: Cross-reference contribution log with ATLAS/AVID/OWASP acknowledgment records. Source: contribution log × external acknowledgments - Corpus recency: Last commit date to the public repository per corpus. Source: public VCS

Answer: - ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets) - ☐ Implemented (Evidence complete + 2 metrics meet targets) - ☐ Partial (Evidence partially complete + <2 metrics meet targets) - ☐ Not Implemented (No industry contributions or open corpus publication)

Evidence Location: __ Validation Date: __ Notes: ___


Question 9: Industry-Shared Exercises

Q9.1: Has the program hosted at least 1 industry-shared red-team exercise per year and participated in ≥2 additional cross-org exercises, with documented cross-org detection-benchmark improvement data from participants?

Evidence Required: - [ ] Exercise log on file: ≥1 hosted exercise per year (OWASP AI chapter, ATLAS practitioner table, or sector ISAC AI red-team); ≥2 additional cross-org exercises participated - [ ] Hosted exercise documented: agenda, participant list, benchmark methodology, measurement of detection improvement before and after the exercise - [ ] Cross-org detection-benchmark improvement data collected from participants and documented in exercise report - [ ] Industry-exercise calendar: next hosted or co-hosted exercise scheduled at least 60 days in advance - [ ] Participation in AISI Inspect evaluation benchmarks, HELM safety evaluations, or sector ISAC AI red-team exercises with anonymized findings on record - [ ] Published regression corpora cited or adopted by at least one external organization; adoption documented

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | Industry-shared exercises per year (hosted + participated) | ___ | ___ | ≥1 hosted + ≥2 participated | ☐ | | | Cross-org detection-benchmark improvement documented | No / Yes | No / Yes | Yes | ☐ | | | Open regression corpora published and maintained upstream | /4 | /4 | ≥4 corpora published | ☐ | | | Industry-shared exercises per year | ___ | ___ | ≥1 hosted + ≥2 participated | ☐ | |

Metric Collection Guidance: - Exercise count: Exercise log filtered to the last 12 months; count hosted and participated exercises. Source: exercise log - Improvement documentation: Hosted exercise report; participant pre/post detection benchmark data. Source: exercise reports - External adoption: External organizations that have cited or forked the public corpora; check repository forks, community citations. Source: public repository analytics + community references - Calendar check: Confirm a specific future exercise date ≥60 days out is on the program calendar. Source: program calendar

Answer: - ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets) - ☐ Implemented (Evidence complete + 2 metrics meet targets) - ☐ Partial (Evidence partially complete + <2 metrics meet targets) - ☐ Not Implemented (No industry-shared exercises conducted or planned)

Evidence Location: __ Validation Date: __ Notes: ___


Summary Scorecard

Level Q# Question Score Weight
L1 Q1 Per-Archetype Test Battery ___
L1 Q2 Regression Corpora in CI ___
L1 Q3 Go-Live Battery and IM Wire ___
L1 Score ___ 0.5
L2 Q4 Tier-Calibrated Battery and Corpus Depth ___
L2 Q5 Scheduled Red-Team Exercises ___
L2 Q6 Cross-Archetype Composition Tests ___
L2 Score ___ 0.3
L3 Q7 Continuous Automated Adversarial Testing Harness ___
L3 Q8 Industry Contributions ___
L3 Q9 Industry-Shared Exercises ___
L3 Score ___ 0.2
Overall ST-Software Score ___

Maturity Level Achieved: ☐ L1 ☐ L2 ☐ L3

Assessment Date: __ Assessor: __ Next Review Date: ___


Document Version: HAIAMM v3.0 Practice: Security Testing (ST) Domain: Software Questionnaire Authored: 2026-05-15 Author: Verifhai

Instructions:

  • Answer based on current practices, not plans
  • “Yes” requires documented evidence
  • Complete all Level 1 questions before Level 2
  • Partial implementation = “No”

↓ Download as Markdown