Security Testing (ST)

Software Domain - HAIAMM v3.0

Practice Overview

Objective: Prove that every AI/HAI software artifact the organization builds behaves correctly under adversarial conditions, by running a foundational per-archetype test battery in CI, maintaining versioned regression corpora, and escalating to scheduled red-team and continuous adversarial testing at higher maturity levels.

Description: ST-Software exercises the AI/HAI software the organization ships, LLM-integrated applications, autonomous agents, RAG pipelines, fine-tuning and training workloads, evaluation harnesses, model-serving services, and classical ML models, against a battery of AI-specific test classes tied directly to the threats in the TA-Software library and the requirements in the SR-Software pack. At L1, every archetype has a published test battery (prompt-injection probes, tool-scope boundary tests, data-egress canaries, logging-completeness tests, kill-switch tests, output-integrity regressions) plus versioned regression corpora (jailbreak corpus, prompt-injection corpus, agent goal-hijack corpus, tool-misuse corpus) running on every PR via CI. L2 adds per-tier scheduled red-team exercises using TA L2's deep threat models, and cross-archetype composition tests. L3 operates continuous automated adversarial testing (AI-vs-AI red-team scaffolding) and contributes findings to MITRE ATLAS, AVID, and OWASP LLM / Agentic Top 10.

Context: Classic CI test suites exercise the happy path and leave the adversarial path untested. An LLM-integrated feature passes all unit tests and then leaks its system prompt on the first prompt-injection attempt. An agent passes its integration tests and then invokes tools outside its declared scope when fed a crafted retrieval response. A fine-tuned model ships to production and regurgitates a canary string from the training corpus. These failures are invisible to classic testing because classic testing was not designed to enumerate AI-specific failure modes, prompt injection (ATLAS TA0001/TA0003), ML model access probing (TA0004), attack staging (TA0012), and data exfiltration via inference (TA0013). ST-Software closes this gap by making AI-specific tests a first-class CI citizen and by connecting them directly to the TA threat library so test coverage tracks threat coverage, not just code coverage.

Maturity Level 1

Objective: Establish a foundational per-archetype test battery and regression corpora that run in CI on every PR, and verify that every AI/HAI software artifact reaches production with a passed go-live battery on record

At this level, the organization gives every AI/HAI software artifact a documented, automated-where-possible test battery drawn from the TA-Software archetype threat library and the SR-Software requirements pack, so every production landing is backed by observable test evidence, not only design assurance.

Dependencies

• TA-Software L1 (required): tests target specific threats from the archetype threat library; without the library, test scope is invented per intake rather than inherited.
• SR-Software L1 (required): requirements pack defines what a passing test means; tests answer "does this artifact meet its SR requirements?" not a free-form question.
• SA-Software L1 (required): reference patterns define the control points (tool allowlist, HITL gate, output filter, secrets vault) that tests exercise.
• IR-Software L1 (required): implementation reviews confirm the artifact is configured as designed before ST probes run; testing an unconfigured artifact produces noise.
• Supports / unblocks: IM-Software L1 (test failures become issues), ML-Software L1 (detections validated by shadow-AI and logging-completeness tests), EH-Software L1 (hardening informed by test findings).

Desired Outcomes

• Every AI/HAI software artifact reaching production has passed a documented per-archetype test battery; the battery result is on file and linked from the SM inventory record.
• Regression corpora (jailbreak, prompt-injection, agent goal-hijack, tool-misuse) run on every PR via CI and catch AI-specific regressions before merge.
• Test failures become IM issues with named owners within one business day, not reports in a drive.
• The battery is automated enough to re-run on demand (post-model-update, post-incident, on cadence) without a bespoke effort per run.
• Test coverage tracks threat coverage: every threat in the TA-Software library for a given archetype has at least one corresponding test in the battery or corpus.

Activities

A) Publish the foundational per-archetype test battery

Ship one test battery per AI/HAI software archetype. Each battery targets the top archetype threats from TA-Software and the archetype-specific SR requirements. L1 target: ≤8 named test classes per archetype. Each test class has: inputs, expected output, pass/fail criteria, evidence artifact (log snippet / trace ID / CI run link), and the TA threat and SR requirement it maps to.

Per-archetype battery (using SM-Software L1 archetypes):

LLM-integrated application or feature: - Prompt-injection probe corpus (regression CI): direct prompt-injection inputs covering system-prompt extraction, instruction-override, role-manipulation, and prefix-injection techniques (ATLAS TA0001/TA0003; AGH); run on every PR against the versioned corpus; failure blocks merge for Critical/High-tier artifacts. - Training-data leakage canary: insert a unique canary string into a fine-tune or RAG corpus; run prefix-completion probes via the inference endpoint; assert the canary is not recoverable (ATLAS TA0013; training-data leakage / OWASP LLM10 partial). - Output-integrity regression: golden test set of 20–50 structured-output prompts with expected schemas; alert on schema drift or content drift exceeding a defined threshold; gate model-version promotions (model-swap regression; ATLAS TA0004). - Rate-limit / abuse-detection test: send inputs at and above the declared rate limit; assert that the rate-limit layer enforces the cap and that abuse-detection fires; verify degraded-mode fallback produces a logged, non-misleading response. - Logging-completeness test: for a known prompt/completion pair, verify that the required log fields (session ID, timestamp, user identifier, prompt hash, completion hash, model version) appear in the org-side log store within the retention SLA.

Autonomous AI agent: - Tool-scope boundary test: call each declared tool with an out-of-scope argument (path traversal, cross-user ID, out-of-range parameter); verify the tool layer rejects the call and logs the rejection (ATLAS TA0004; EA / TM). - Tool-argument-smuggling corpus (regression CI): versioned corpus of argument-smuggling payloads (path traversal fragments, SQL meta characters, JSON-injection strings, user-ID substitution patterns); run on every PR; assert rejection at validation layer (TM; OWASP LLM07). - Multi-turn agent goal-hijack (AGH) probe corpus: seed the agent's retrieval path or tool-response payload with alternate-goal instructions; verify the agent does not change its declared goal; run on every PR (ATLAS TA0001/TA0003; AGH). - Recursive-invocation guardrail test: craft input that causes the agent to attempt a recursive or looping tool invocation; verify the recursion budget is enforced before cost or blast-radius exceeds the declared SLA (TM / RA). - Kill-switch test: trigger the kill-switch path under a simulated load condition; verify the agent process halts within the declared SLA (≤5 minutes); log the halt event; assert that in-flight tool invocations are not silently completed after kill signal (RA). - HITL-gate test: invoke a tool in the "destructive / external-network / customer-account-affecting" category; verify the human-confirmation gate fires and that the action cannot complete without an explicit approval event in the log (EA / OWASP LLM08). - Session-bound memory test: run a long session (100+ turns); verify that cross-session memory does not persist regulated or customer-specific data past the declared session boundary; assert the per-session token budget is enforced (RA).

RAG / retrieval-augmented generation pipeline: - Retrieval-source provenance test: query the pipeline with a known-origin document in the corpus; verify the retrieved chunk carries the correct source identifier and classification label through to the model response log (retrieval poisoning; ATLAS TA0003). - Classification-label-respecting retrieval test: with a query that would match a document classified "Confidential" and a query context that is "Public," assert the confidential chunk is not returned or is filtered before reaching the prompt (cross-classification leakage; EA). - Injection-defense test over retrieved content: seed the corpus with a document containing prompt-injection instructions; verify the agent or app does not follow those instructions; assert the provenance label marks the chunk as "retrieved / untrusted" (AGH; OWASP LLM01 indirect). - Per-tenant retrieval isolation test: with a multi-tenant deployment, send a query from tenant A that would match a document indexed under tenant B's namespace; verify zero results or a namespacing error, not tenant B's data, is returned (EA; cross-tenant leakage).

Fine-tuning or model-training workload: - Training-data classification pre-flight test: before any fine-tuning job starts, run a data-classification check against the proposed training dataset; assert that regulated data classes (PII, PHI, PCI, customer source code) fail the pre-flight unless a documented privacy sign-off is on file (training-data leakage; ATLAS TA0002). - No-regulated-PII-without-sign-off test: attempt to queue a fine-tuning job with a synthetic dataset containing a known PII pattern (e.g., SSN-format strings); assert the job is blocked at intake without a valid privacy-officer approval artifact. - Model-card auto-population test: on completion of a fine-tuning job, assert that the model registry record carries: training data sources, data-class classification, eval-suite results, and the date/identity of the approving reviewer; fail the promotion gate if any field is empty. - Eval-suite gating test: confirm that a model cannot be promoted from the training registry to the serving registry without a passing eval-suite result on the defined eval harness; attempt a promotion without the eval artifact and assert it is blocked (output-integrity; ATLAS TA0004).

Evaluation / red-team harness: - Isolation test: assert that eval data does not flow to training pipelines; verify via data-flow check that the eval harness's input datasets and annotated outputs are not accessible from the training workload's artifact store. - Reproducibility test: run the same eval suite twice against the same pinned model version and assert the results are within the declared variance threshold; fail if results differ beyond threshold (output-integrity regression). - Regression-corpus completeness test: assert that the eval harness includes test cases from the TA-Software archetype library for the artifact under evaluation; fail if the coverage ratio falls below the declared target (e.g., all top-5 archetype threats covered).

Model-serving service: - Version-pinning test: assert that the serving endpoint returns the declared model version in the response metadata; attempt to serve a request without a pinned version specified and assert the request is rejected or routed to the default-pinned version, not an arbitrary latest. - Canary-deployment test: assert that a new model version is deployed to a canary slice before full rollout; verify that rollout automation checks the canary eval result before promoting to full traffic. - Rollback test: trigger the rollback playbook for a canary deployment; verify the serving endpoint reverts to the previous pinned version within the declared rollback SLA; assert the rollback event is logged.

Classical ML model in a product surface: - Drift-detection threshold test: inject a synthetic distribution-shift into the model's input stream; assert the drift-detection signal fires within the declared detection window. - Retraining-trigger test: verify that when drift-detection fires, the defined retraining or human-review trigger is activated and produces an IM ticket with the correct severity tag.

B) Build and maintain regression corpora in CI

Maintain four versioned regression corpora in source control, running on every PR via CI. Each corpus is a collection of structured test fixtures: input, expected safe output pattern, threat tag (HAI TTP + ATLAS tactic ID), OWASP reference, source, date added.

• Jailbreak corpus, 30–100 jailbreak inputs targeting role-override, persona-switch, authority-claim, and encoding-bypass patterns; run against all LLM-integrated app and agent archetypes; failure blocks merge for Critical/High-tier.
• Prompt-injection corpus, 30–100 direct and indirect prompt-injection inputs (direct: system-prompt extraction, instruction override; indirect: injected-document payloads, tool-response payloads, multi-turn history injection); run against all LLM-integrated, agent, and RAG archetypes.
• Agent goal-hijack (AGH) corpus, 20–60 multi-turn sequences designed to redirect a tool-using agent's declared goal via retrieval-path content, tool responses, or long-context accumulation; run against agent archetypes.
• Tool-misuse corpus, 20–60 argument-smuggling and out-of-scope invocation payloads per tool type (file-path tools, HTTP tools, database-query tools, API-call tools); run against agent archetypes.

Corpus management: - Versioned in source control; corpus changes go through PR review with a named corpus owner. - Corpus refresh cadence: monthly minimum from three sources, internal observations (IR findings, IM incidents, red-team results), external public corpora (OWASP LLM Top 10 examples, HackAPrompt dataset, ATLAS technique examples), and jailbreak research repositories. - CI run budget-capped (token spend limit per run); failing runs are a blocking CI check for Critical/High-tier; non-blocking warning for Medium/Low. - New PR adding or modifying AI/HAI software that changes archetype triggers a corpus completeness check before merge.

C) Operate the go-live battery and wire test failures to IM

Every AI/HAI software artifact must pass its archetype battery before receiving Sanctioned status in the SM inventory. Go-live triggers: - Pre-production: all applicable archetype tests must pass before the artifact is promoted to production; the go-live test record is linked from the SM inventory and the PC intake artifact. - Post-model-update: any model-family swap, model-version promotion, or new fine-tune triggers a re-run of the output-integrity regression and the full archetype battery for the serving layer within 14 days (Critical-tier: within 7 days). - Post-incident: any IM-Software incident involving the artifact triggers a re-run of the relevant battery subset before the incident is closed. - Quarterly: all active AI/HAI software artifacts re-run their battery; results reviewed by the named test-battery owner.

All test failures route to IM-Software within one business day with a severity tag derived from the ST severity rubric (Blocker / Critical / High / Medium / Low). Named battery owner per archetype; battery ownership is a named role, not a shared-team responsibility.

Outcome Metrics (L1)

Process Metrics (leading)

• Battery owner named per archetype; quarterly re-run scheduled in advance.
• Test-automation coverage, target ≥60% of battery items running in CI without human intervention at L1.
• Corpus refresh cadence honored monthly; corpus owner reviews incoming sources.
• New-archetype lead time, from "first intake in new software category" to "battery published" ≤30 days.

Effectiveness Metrics (business value)

• Pre-production catch rate, test failures that uncovered a misconfiguration or control gap before production landing (prompt injection in new agent, tool-scope overshoot in RAG, system-prompt leakage in new LLM-integrated feature).
• Model-swap regression catch rate, output-integrity regressions caught by the battery before the model version shipped to full traffic.
• Reduced IM incident volume, artifacts with a full battery pass-rate in CI have a measurably lower IM incident rate than those without (tracked as a rolling 12-month comparison).

Success Criteria

• Per-archetype foundational test battery published for all seven archetypes (LLM-integrated app, agent, RAG, fine-tune/training, eval harness, model-serving service, classical ML), linked from the SM inventory record and the DR/IR artifacts.
• Four regression corpora (jailbreak, prompt-injection, AGH, tool-misuse) published in source control, running in CI on every PR for Critical/High-tier artifacts, with a named corpus owner and a monthly refresh cadence.
• 100% of AI/HAI software artifacts reaching production in the last 90 days have a passed go-live battery on record.
• All test failures routed to IM with a 1-day handoff SLA and named owner.
• Named battery owner per archetype; CI automation covers ≥60% of battery items.

Maturity Level 2

Objective: Calibrate test depth per risk tier using the SM L2 tier-treatment matrix, run scheduled red-team exercises per tier using TA L2 deep threat models, and test cross-archetype compositions for Critical-tier artifacts

At this level, testing stops treating all AI/HAI software artifacts the same. Per-tier calibration drives what each artifact receives from the test program. Critical-tier artifacts are red-teamed quarterly; High-tier semi-annually. TA L2 deep threat models replace archetype snapshots as the scenario library for red-team exercises. Cross-archetype compositions (e.g., a customer-facing agent that also does RAG and calls a fine-tuned model) receive combined test suites that exercise composition-specific failure modes.

Dependencies

• ST-Software L1 (required): per-archetype batteries and regression corpora must be operational before per-tier calibration is meaningful.
• SM-Software L2 (required): the risk-tier rubric and tier-treatment matrix determine which artifacts receive full-scope testing (Critical/High) vs. a subset (Medium/Low); without tier assignments, L2 calibration has no substrate.
• TA-Software L2 (required): per-artifact deep threat models for Critical-tier artifacts shape red-team scope; archetype-only snapshots are insufficient for L2 red-team exercises.
• Supports / unblocks: IM-Software L2 (tier-calibrated incident handling relies on tier-calibrated test signals), ML-Software L2 (detections tuned per-artifact threat model depend on L2 test findings to validate coverage).

Desired Outcomes

• Test depth is visibly differentiated: Critical-tier artifacts get the full battery plus quarterly red-team and cross-archetype composition tests; Low-tier get the base CI corpus and a spot-check.
• Red-team findings are scenario-based (from TA L2 per-artifact threat models), not free-form; findings trace to specific ATLAS tactics and to SR requirements.
• The regression corpora grow from red-team findings: every critical or high-severity red-team finding produces a new corpus entry that runs in CI within 30 days.
• Cross-archetype composition failure modes (agent + RAG interaction, fine-tune + serving-service interaction, eval harness → training feedback loop) are in scope and have documented test coverage.
• Per-tier SLA adherence for testing activities is tracked and reported to the program sponsor.

Activities

A) Tier-calibrated test battery and CI corpus depth

Publish a per-tier test treatment aligned to SM L2's tier-treatment matrix:

For Critical and High-tier artifacts, the CI regression corpus is separated into: - A Critical-tier corpus running on every PR (30–100 entries, tuned to the artifact's specific tool set, retrieval sources, and data classes from the TA L2 per-artifact threat model). - A broader tier-appropriate corpus running on merge.

B) Scheduled per-tier red-team exercises using TA L2 threat models

Red-team cadence by tier: - Critical: quarterly (4 per year); scope derived from TA L2 per-artifact deep threat model; covers prompt-injection chains, indirect prompt injection via RAG retrieval, agent tool abuse, jailbreak regression, data-egress canaries, multi-turn AGH probes, cross-tenant isolation probes. - High: semi-annual (2 per year); scope from TA L2 artifact deltas; covers the top-5 threats from the per-artifact model. - Medium/Low: ad-hoc (before major model changes or scope expansions); archetype snapshot drives scope.

Each red-team exercise follows the AI Security Testing Methodology §9: written rules of engagement, test plan reviewed with artifact owner, execution log, structured findings report (severity / root cause / ATLAS tactic ID / SR requirement traced), remediation pairings. Red-team findings at Critical or High severity produce corpus entries within 30 days.

Cross-archetype composition tests for Critical-tier: - Agent + RAG composition: indirect-injection via RAG path into agent goal-hijack chain. - Fine-tune + model-serving: membership-inference probe against the served fine-tuned model; canary extraction from production completions. - Multi-agent orchestration: sub-agent goal-hijack via orchestrator response; scope inheritance tests (sub-agent cannot exceed parent scope).

C) Red-team findings → corpus pipeline

Every Critical or High-severity red-team finding produces: 1. A new corpus entry (input, expected safe output, threat tag, ATLAS tactic ID, date, source reference) committed to the relevant regression corpus within 30 days. 2. An IM-Software finding with severity tag and the named artifact owner as assignee. 3. A TA-Software library-gap ticket if the finding was not in the archetype library, the gap is tracked with a named owner and a 30-day close SLA for Critical-tier gaps.

Outcome Metrics (L2)

Process Metrics (leading)

• Red-team schedule on calendar; no Critical-tier artifact skips a quarterly exercise.
• Corpus review cadence, monthly.
• Cross-archetype composition test plan published for each Critical-tier artifact with composition; reviewed by named architect.
• Finding → TA library-gap pipeline: Critical gaps closed within 30 days; High within 60 days.

Effectiveness Metrics (business value)

• Pre-production catch rate for Critical-tier increases as red-team exercises catch issues before model-update deploys.
• Regression corpus catches model-update regressions early, model-swap regressions that would have reached production users, detected by CI corpus before full traffic rollout.
• Incident rate for Critical-tier artifacts with full-scope testing is measurably lower than for those without (rolling 12-month comparison).

Success Criteria

• Quarterly red-team for 100% of Critical-tier artifacts; semi-annual for 100% of High-tier; scope tied to TA L2 per-artifact deep threat models.
• Critical-tier regression corpora (all 4) running on every PR for all Critical-tier artifacts; per-tier calibration enforced.
• ≥90% of Critical/High-severity red-team findings converted to corpus entries within 30 days.
• Cross-archetype composition tests documented and run for all Critical-tier artifacts with composite archetypes.
• Per-tier SLA adherence for testing activities ≥90%.

Maturity Level 3

Objective: Operate continuous automated adversarial testing for Critical-tier artifacts, publish regression corpora and findings as open artifacts, and contribute discovered TTPs to MITRE ATLAS, AVID, and OWASP LLM / Agentic Top 10

At this level, testing runs continuously rather than periodically. An automated adversarial testing harness, AI-vs-AI red-team scaffolding, probes Critical-tier artifacts daily using generated jailbreak ladders, indirect-injection seeders, and tool-misuse generators. Novel findings are triaged into the TA library weekly. Anonymized regression corpora, test patterns, and discovered TTPs are contributed to MITRE ATLAS, AVID, OWASP LLM/Agentic Top 10, and external benchmarks (HELM, AISI Inspect evaluations, sector ISAC AI working groups).

Dependencies

• ST-Software L2 (required): per-tier red-team function, regression corpora, and finding → corpus pipeline must be operational.
• TA-Software L3 (required): automated TTP-ingestion pipeline from ST telemetry feeds the TA L3 auto-update loop.
• ML-Software L2+ (required): detections instrumented to catch automated-probe activity so the harness and the monitoring pipeline are calibrated against each other.
• SM-Software L3 (alignment): automated inventory signals feed the automated test harness with the current Critical-tier artifact list and their tier metadata.

Desired Outcomes

• Critical-tier adversarial posture is measured daily, not quarterly.
• Novel TTPs discovered by the automated harness reach the TA library within 14 days, the org's defenses update faster than most peers' quarterly red-team cycles.
• Program-originated findings appear in MITRE ATLAS, AVID, and OWASP revisions, the org is a net contributor to the AI software security ecosystem.
• Published regression corpora and test patterns are adopted by peer organizations, demonstrating industry-leading testing maturity.

Activities

A) Continuous automated adversarial testing harness

Deploy an AI-vs-AI automated red-team scaffolding that runs daily against all Critical-tier AI/HAI software artifacts: - Prompt-injection generator: produces novel direct and indirect prompt-injection inputs using mutation of the regression corpus, template-based variation, and model-assisted jailbreak-ladder generation; runs against LLM-integrated app, agent, and RAG archetypes. - Indirect-injection seeder: generates poisoned-document payloads designed to redirect agent goals; seeds a test corpus environment and exercises the retrieval → prompt → action pipeline. - Tool-misuse generator: generates argument-smuggling variants for each declared tool in each Critical-tier agent; probes argument validation boundaries and unexpected tool combinations. - Output-integrity monitor: runs the golden test set daily against production endpoints; any drift triggers a P1 alert to IM-Software and a re-run of the full regression corpus.

Findings triaged by a named ST owner at least weekly. New TTPs (patterns not in the TA library) fed into the TA L3 auto-proposal pipeline within 14 days. High-severity automated findings route to IM-Software within 24 hours.

B) Contribute findings to industry

Contribute anonymized, legally-vetted findings to: - MITRE ATLAS, new technique observations (novel prompt-injection variants, agent-loop attack patterns, new retrieval-poisoning mechanics); submissions follow ATLAS evidence-and-provenance requirements; target ≥4 contributions per year. - AI Vulnerability Database (AVID), structured disclosure submissions for novel vulnerabilities in own-built AI/HAI software or its upstream dependencies (coordinated disclosure where third-party components are involved). - OWASP LLM Top 10 / Agentic Top 10, real-world telemetry evidence during revision cycles; target ≥2 substantive submissions per revision cycle. - External benchmarks, participate in AISI Inspect evaluation benchmarks, HELM safety evaluations, and sector ISAC AI red-team exercises with anonymized findings.

C) Publish regression corpora and test patterns as open artifacts

• Publish anonymized versions of the four regression corpora (jailbreak, prompt-injection, AGH, tool-misuse) under an open license; scrubbed of org-specific tool names, data classes, and artifact identifiers.
• Maintain the published versions upstream; internal corpora are a superset of the published versions with org-specific entries not shared externally.
• Host or co-host at least one industry red-team benchmark per year (OWASP AI chapter, ATLAS practitioner table, sector ISAC AI working group); collect cross-org detection-benchmark improvement data.

Outcome Metrics (L3)

Process Metrics (leading)

• Continuous harness health, % Critical-tier artifacts producing a fresh automated probe result within the last 24 hours; on-call paged when a harness feed goes stale >24 hours.
• New-TTP triage cadence, automated probe findings reviewed weekly by named ST owner; novel patterns forwarded to TA library within 14 days.
• Industry-contribution pipeline, at least one anonymized finding in-preparation, in-legal-review, or submitted at any time.
• Industry-exercise calendar, next hosted or co-hosted exercise scheduled at least 60 days in advance.

Effectiveness Metrics (business value)

• Mean time to detect novel AI-specific attack techniques decreases as the continuous harness catches model-update regressions within hours, not sprint cycles.
• Program-originated TTPs recognized in MITRE ATLAS or AVID demonstrate external validation of testing rigor.
• Cross-org exercise participants cite improved detection benchmarks from shared corpora, measurable uplift in peer organizations' defenses.
• Critical-tier incidents attributable to prompt injection, tool-scope abuse, or data-egress drop as the continuous harness closes the gap between quarterly red-team cycles.

Success Criteria

• ≥80% of Critical-tier AI/HAI software artifacts under continuous automated adversarial testing with daily probe execution.
• New-TTP ingestion lead time ≤14 days; automated findings triaged weekly by named owner.
• ≥4 industry contributions per year to MITRE ATLAS / AVID / OWASP LLM / Agentic Top 10.
• ≥4 open regression corpora published under a permissive license and maintained upstream.
• ≥1 industry-shared exercise hosted per year plus ≥2 participated; cross-org detection-benchmark improvement documented.

Key Success Indicators

Level 1: - Per-archetype foundational test battery published for all seven archetypes (LLM-integrated app, agent, RAG, fine-tune/training, eval harness, model-serving service, classical ML), each test class tied to a TA-Software archetype threat and an SR-Software requirement. - Four regression corpora (jailbreak, prompt-injection, AGH, tool-misuse) in source control, running on every PR for Critical/High-tier artifacts, with named corpus owner and monthly refresh cadence. - 100% of AI/HAI software artifacts reaching production in the last 90 days have a passed go-live battery on record linked from the SM inventory. - All test failures routed to IM within 1 business day; CI automation covers ≥60% of battery items. - TA-Software archetype threat coverage by test battery ≥80%.

Level 2: - 100% of Critical-tier artifacts red-teamed quarterly; 100% of High-tier semi-annually; scope tied to TA L2 per-artifact deep threat models. - Per-tier calibration enforced: Critical-tier gets all 4 corpora on every PR plus daily output-integrity regression; Low-tier gets jailbreak corpus on merge. - ≥90% of Critical/High-severity red-team findings converted to corpus entries within 30 days. - Cross-archetype composition tests documented and run for all Critical-tier composite artifacts.

Level 3: - ≥80% of Critical-tier artifacts under continuous automated adversarial testing with daily probe execution; novel TTPs reaching TA library within 14 days. - ≥4 industry contributions per year to MITRE ATLAS / AVID / OWASP LLM / Agentic Top 10; ≥4 open regression corpora published and maintained. - ≥1 hosted + ≥2 participated industry exercises per year with documented cross-org detection-benchmark improvement.

Common Pitfalls

Level 1: - ❌ Test battery reduced to a logging-completeness check and a golden-path assertion, no behavioral adversarial probes (prompt injection, tool-scope boundary, kill-switch) actually exercised. - ❌ Regression corpora committed to source control but not wired into CI, they exist but run only when a reviewer manually triggers them; coverage erodes after every sprint. - ❌ Go-live battery runs once pre-production but is never re-run after model-version updates, test coverage erodes as model versions change and configurations drift. - ❌ Test failures logged in a spreadsheet separate from IM, no SLA enforcement, no aging visibility, no named owner; the same failure recurs across multiple PRs undetected. - ❌ Corpus refresh skipped for three months because "nothing new happened", public jailbreak research and ATLAS technique additions pile up unread; the corpus is frozen at launch while the external threat landscape advances. - ❌ Test battery conflates agent and LLM-integrated app, agent-specific tests (kill-switch, HITL gate, tool-scope boundary) are absent from the agent battery because the same checklist was applied to both archetypes. - ❌ Training-data canary planted at fine-tune time but never probed via the serving endpoint, the leakage test is half-implemented; the canary is there but extraction is never attempted.

Level 2: - ❌ Red-team scope defined as "prompt-injection probes" but indirect-prompt-injection via RAG retrieval, argument smuggling, multi-turn AGH chains, and cross-tenant isolation tests are excluded, the top threat classes for agent + RAG compositions go untested. - ❌ Corpus growth declared at ≥1 per month but entries are variations on the same payload format, the corpus does not cover the new jailbreak techniques published since L2 launch; model-update regressions pass the corpus and fail in production. - ❌ Per-tier calibration documented in the tier-treatment matrix but CI pipeline applies the same corpus to all tiers, Critical artifacts run the same tests as Low; differentiation exists on paper only. - ❌ Red-team findings route to IM but the finding → corpus pipeline is never executed, 12 months of Critical/High findings sit in IM as closed tickets with no corpus entries; the same vulnerabilities are re-discovered at the next red-team exercise. - ❌ Cross-archetype composition tests scoped but not executed because "no engineer owns agent-plus-RAG testing", composition-specific failure modes (indirect injection via RAG into agent goal-hijack chain) are in the threat model but not in any test.

Level 3: - ❌ Continuous harness runs prompt-injection probes that the artifact's content filter trivially blocks, coverage metric looks good but the probes are not exercising the real threat surface; novel jailbreak techniques are not generated. - ❌ Industry contributions are legal-vetted case-study summaries rather than actionable, reproducible technique descriptions, ATLAS reviewers cannot map them to a technique ID; AVID submissions lack reproducibility notes. - ❌ Open corpora published once and then not maintained, external organizations build on a stale version while the internal corpus has 60 new entries; discrepancies surface at community exercises and damage the program's credibility. - ❌ New-TTP ingestion from automated probes to the TA library is manual and quarterly, by the time a novel technique reaches SR and SA updates and is reflected in controls, the technique is already being exploited in the wild. - ❌ Hosted industry exercise becomes a capabilities showcase rather than a detection-benchmarking session, no measurable improvement data is collected from participants; the "≥1 hosted per year" metric is met without producing any cross-org security uplift.

Practice Maturity Questions

Level 1: 1. Is a per-archetype foundational test battery published for all seven AI/HAI software archetypes, with each test class tied to a TA-Software archetype threat (HAI TTP + ATLAS tactic ID) and an SR-Software requirement, defined inputs/outputs/pass-fail criteria, and an evidence artifact, and are 100% of new AI/HAI software artifacts required to pass the battery before production Sanctioned status is issued? 2. Are four regression corpora (jailbreak, prompt-injection, AGH, tool-misuse) versioned in source control, running in CI on every PR for Critical/High-tier artifacts, with a named corpus owner, a monthly refresh cadence from internal + external sources, and a CI token-spend budget cap, and are ≥95% of Critical/High-tier PR merges verified to have run and passed the corpus? 3. Are all test failures routed to IM-Software within 1 business day with a severity tag and named owner, and does TA-Software archetype threat coverage by the test battery and corpus reach ≥80% by end of year one?

Level 2: 1. Are 100% of Critical-tier AI/HAI software artifacts red-teamed at least quarterly, and 100% of High-tier semi-annually, with scope derived from TA-Software L2 per-artifact deep threat models, covering prompt-injection chains, indirect-prompt-injection via RAG, agent tool abuse, multi-turn AGH probes, data-egress canaries, and cross-tenant isolation, with findings routed to IM and remediation tracked? 2. Is per-tier corpus calibration enforced in CI (Critical-tier: all 4 corpora on every PR + daily output-integrity regression; Low-tier: jailbreak corpus on merge), and are ≥90% of Critical/High-severity red-team findings converted to corpus entries within 30 days? 3. Are cross-archetype composition tests (agent + RAG, fine-tune + model-serving, multi-agent orchestration) documented and executed for all Critical-tier composite artifacts, and is per-tier SLA adherence for testing activities ≥90%?

Level 3: 1. Are ≥80% of Critical-tier AI/HAI software artifacts under continuous automated adversarial testing with daily probe execution, with novel TTPs triaged into the TA-Software library within 14 days and high-severity automated findings routed to IM within 24 hours? 2. Has the program contributed ≥4 anonymized, legally-vetted findings per year to MITRE ATLAS, AVID, or OWASP LLM/Agentic Top 10, with at least one accepted as a new or refined technique, and are ≥4 open regression corpora published under a permissive license and maintained upstream? 3. Has the program hosted at least 1 industry-shared red-team exercise per year and participated in ≥2 additional cross-org exercises, with documented cross-org detection-benchmark improvement data from participants?

Document Version: HAIAMM v3.0 Practice: Security Testing (ST) Domain: Software Last Updated: 2026-05-13 Author: Verifhai