# Security Testing (ST)
## Endpoints Domain - HAIAMM v3.0

---

### Practice Overview

**Objective:** Prove that every AI/HAI-enabled endpoint and user-facing AI interface **behaves correctly under adversarial conditions**, by running a foundational per-archetype test battery, maintaining versioned regression corpora, and escalating to scheduled per-tier red-team and continuous adversarial testing at higher maturity levels.

**Description:** ST-Endpoints exercises the AI/HAI endpoint deployments the organization *operates and consumes*, AI assistants on managed endpoints, browser-based AI tools, chatbots and conversational UIs, multi-modal AI interfaces, SaaS-AI productivity features, mobile AI apps, and edge AI devices, against a battery of AI-specific test classes tied directly to the threats in the TA-Endpoints library and the requirements in the SR-Endpoints pack. At L1, every archetype has a published test battery plus versioned regression corpora (prompt-injection corpus, jailbreak corpus, multi-modal injection corpus, DLP-paste-block corpus, browser-extension-scope corpus, Art. 50 disclosure corpus) running on deployment and periodically thereafter. L2 adds per-tier red-team exercises (Critical quarterly; High semi-annually) and cross-archetype composition tests (assistant + browser extension; chatbot + multi-modal). L3 operates continuous automated adversarial testing and contributes findings to MITRE ATLAS, AVID, OWASP MASVS, OWASP Browser-Extension Top 10, and CSA endpoint working groups.

**Context:** Classic acceptance test suites exercise the happy path and leave the adversarial path untested. A chatbot passes all functional tests and then follows an injected instruction from a user-crafted input. A browser extension passes its scope review and then reads pages it was not designed to access when a crafted page exploits an over-broad host permission. An AI assistant on a managed endpoint bypasses the tool-allowlist because the enforcement logic was never wired in the deployed configuration. These failures are invisible to classic testing because classic testing was not designed to enumerate AI-specific failure modes, prompt injection (ATLAS TA0001/TA0003), output-integrity regression (TA0004), data exfiltration via inference (TA0013), and physical-interface attacks (edge AI devices). ST-Endpoints closes this gap by making AI-specific endpoint tests a first-class validation citizen and connecting them directly to the TA threat library so test coverage tracks threat coverage, not just configuration coverage.

---

## Maturity Level 1
### Objective: Establish a foundational per-archetype test battery and regression corpora for AI/HAI-enabled endpoints, and verify that every endpoint reaches production with a passed go-deployment battery on record

At this level, the organization gives every AI/HAI-enabled endpoint a documented, automated-where-possible test battery drawn from the TA-Endpoints archetype threat library and the SR-Endpoints requirements pack, so every production deployment is backed by observable test evidence, not only design assurance.

#### Dependencies

- **TA-Endpoints L1 (required):** tests target specific threats from the archetype threat library; without the library, test scope is invented per deployment rather than inherited.
- **SR-Endpoints L1 (required):** requirements pack defines what a passing test means; tests answer "does this endpoint meet its SR requirements?" not a free-form question.
- **SA-Endpoints L1 (required):** reference patterns define the control points (DLP scope, extension allowlist, output filter, kill-switch, Art. 50 disclosure) that tests exercise.
- **IR-Endpoints L1 (required):** implementation reviews confirm the endpoint is configured as designed before ST probes run; testing a misconfigured endpoint produces noise.
- **Supports / unblocks:** IM-Endpoints L1 (test failures become issues), ML-Endpoints L1 (detections validated by logging-completeness tests), EH-Endpoints L1 (hardening informed by test findings).

#### Desired Outcomes

- Every AI/HAI-enabled endpoint reaching production has passed a documented per-archetype test battery; the battery result is on file and linked from the SM-Endpoints inventory record.
- Regression corpora (prompt-injection, jailbreak, multi-modal injection, DLP-paste-block, browser-extension-scope, Art. 50 disclosure) run on deployment and periodically thereafter, catching AI-specific regressions before they reach users.
- Test failures become IM-Endpoints issues with named owners within one business day, not reports in a drive.
- The battery is automated enough to re-run on demand (post-model-update, post-incident, on cadence) without a bespoke effort per run.
- Test coverage tracks threat coverage: every threat in the TA-Endpoints library for a given archetype has at least one corresponding test in the battery or corpus.

#### Activities

**A) Publish the foundational per-archetype test battery**

Ship one test battery per AI/HAI endpoint archetype. Each battery targets the top archetype threats from TA-Endpoints and the archetype-specific SR requirements. L1 target: ≤8 named test classes per archetype. Each test class has: inputs, expected output, pass/fail criteria, evidence artifact, and the TA threat and SR requirement it maps to.

**AI assistant / copilot on managed endpoint:**
- *DLP-paste-block test:* paste a regulated-data canary string (credit card pattern, SSN-format string, synthetic PHI) into the AI assistant's input field; verify the DLP policy blocks or alerts on the transfer before the content reaches the AI vendor model; record the block / alert event from the DLP log (SR DLP requirement; TA-Endpoints data-exfiltration threat).
- *Managed-endpoint requirement test:* attempt to install and use the AI assistant on a non-MDM-enrolled device; verify the assistant cannot authenticate or launch without enrollment; record the rejection event (SA-Endpoints managed-endpoint pattern; TA identity threat).
- *Tool-allowlist test:* configure the assistant to attempt a local action not on the declared tool allowlist (e.g., a file-system path outside the declared scope, a local API call not listed); verify the action is blocked at the allowlist enforcement layer and logged (SR tool-allowlist requirement; ATLAS TA0004).
- *Audit-log completeness test:* issue a known invocation and verify that the required log fields (user identity, timestamp, invocation content hash, outcome) appear in the audit log within the declared retention SLA.

**Browser-based AI tool:**
- *Extension-scope test:* load the browser extension on a page outside its declared host-permission scope; verify the extension cannot read or modify the page's DOM; record the permission rejection (OWASP Browser-Extension Top 10; SR scope-minimization requirement).
- *DLP-integration test:* enter a regulated-data canary into a field on a monitored page while the extension is active; verify the browser DLP policy blocks or alerts (SR DLP requirement; TA data-exfiltration).
- *Backend-SSO test:* attempt to authenticate with the browser tool using a non-SSO credential or an expired session; verify the tool rejects the request and redirects to org SSO (SR identity requirement; TA credential threat).
- *Extension-injection test:* inject a crafted script into a page that the extension processes; verify the extension does not execute the injected code or pass it to the backend AI model as instructions (ATLAS TA0001/TA0003; prompt-injection via extension surface).

**Chatbot / conversational UI:**
- *Prompt-injection regression CI corpus:* direct and indirect prompt-injection inputs covering system-prompt extraction, instruction-override, role-manipulation, and multi-turn injection; run against the deployed chatbot on every deployment update; failure blocks go-live for Critical/High-tier (ATLAS TA0001/TA0003).
- *Jailbreak regression corpus:* jailbreak inputs targeting role-override, persona-switch, authority-claim, and encoding-bypass; run on every deployment update.
- *Data-exfiltration probe corpus:* inputs designed to extract other users' data, the system prompt, or training data from the model; verify the chatbot does not expose cross-user data, system-prompt content, or training-corpus fragments (ATLAS TA0013; OWASP LLM06).
- *Art. 50 disclosure-presence test:* send an opening message to the live chatbot and verify the AI disclosure is present in the response or in the UI at the time of the first interaction; test on every deployment update and on every UX release; failure is a Critical finding (EU AI Act Art. 50; SR disclosure requirement).
- *Escalation-path test:* request escalation to a human agent within the chatbot; verify the customer reaches a human within the declared SLA; record the escalation event and the time-to-human measurement.

**Multi-modal AI interface:**
- *Image-injection corpus (steganographic prompts):* submit images embedding encoded prompt-injection instructions; verify the interface does not follow the embedded instructions; run against the deployed interface on every deployment update (ATLAS TA0001 indirect; multi-modal injection).
- *Voice-injection corpus:* submit audio clips encoding adversarial instructions (supersonic embedding, instruction whisper over noise); verify the interface does not follow embedded instructions (ATLAS TA0001 indirect; voice-channel injection).
- *Deepfake-detection test:* submit a synthetically generated face or voice to the interface; verify the interface's declared deepfake-detection mechanism flags or rejects the synthetic content (SR modality-integrity requirement).
- *Cross-modal consistency test:* submit the same intent via text, image, and voice modalities; verify the responses are semantically consistent and all pass the output safety filter; a response that is safe via one modality but unsafe via another is a Critical finding.

**AI-augmented productivity (SaaS-AI feature on endpoint):**
- *Silent-enablement detection test:* verify the admin-audit log captures an AI feature enablement event when a SaaS-AI feature is enabled tenant-wide; trigger a test enablement in a non-production tenant and confirm the IR-Endpoints drift detection receives the event within the declared detection window (SR admin-audit requirement; TA silent-enablement threat).
- *Data-scope test:* verify the SaaS-AI feature cannot access workspace content outside its DR-approved scope; in a test tenant, configure the feature and attempt to access a workspace section explicitly excluded in the DR record, confirm the feature cannot retrieve or process the excluded content (SR data-scope requirement; OWASP LLM02 data-leakage).
- *Regulated-data-flow test:* enter a regulated-data canary into a monitored workspace field and invoke the SaaS-AI feature; verify the DLP policy blocks or alerts on the regulated data flowing to the vendor AI model before it is processed (SR DLP requirement; TA data-exfiltration).

**Mobile AI app:**
- *App-signature verification test:* install a test build with a modified signature; verify the app's signature verification mechanism rejects the modified build and does not allow the AI feature to run (SR app-integrity requirement; TA supply-chain threat).
- *Local-model integrity test:* replace the on-device model file with a modified version; verify the app detects the hash mismatch and refuses to load the modified model (SR model-integrity requirement; ATLAS TA0002).
- *Permission-minimization test:* review the installed app's declared permissions against the minimum declared in the DR checklist; flag any permission present in the installed build that is absent from the DR-approved set (SR permission-minimization requirement).
- *Biometric-bypass-via-AI test:* attempt to authenticate to the app's AI feature using a synthetically generated face or voice that impersonates the enrolled user; verify the authentication mechanism rejects the synthetic input (SR biometric-integrity requirement).

**Edge AI device:**
- *Firmware signature test:* attempt to flash a firmware image with an invalid or unsigned signature; verify the device rejects the unsigned firmware and remains on the current validated firmware (SR firmware-integrity requirement; TA supply-chain threat).
- *Model signature test:* attempt to load a model file with an invalid or modified hash; verify the device refuses to load the modified model (SR model-integrity requirement; ATLAS TA0002).
- *Physical-tamper test:* trigger the device's physical-tamper detection mechanism (using the manufacturer's declared test method or a simulation); verify the tamper event is recorded and reported to the management plane (SR tamper-detection requirement).
- *Uplink encryption test:* capture network traffic from the device and verify the uplink uses the DR-approved cipher suite and TLS version; a downgrade from the approved configuration is a Critical finding (SR encryption requirement).
- *Remote-disable test:* issue the remote-disable command via the device management plane; verify the device's AI capability becomes unavailable within the declared SLA; record the test date, command timestamp, and disable-confirmation time (SR kill-switch requirement; TA excessive-agency threat).

**B) Build and maintain regression corpora**

Maintain six versioned regression corpora, run on deployment and on a periodic cadence. Each corpus is a collection of structured test fixtures: input, expected safe output pattern, threat tag (HAI TTP + ATLAS tactic ID), OWASP reference, source, date added.

- **Prompt-injection corpus**, 30–100 direct and indirect prompt-injection inputs for chatbot, AI assistant, and multi-modal archetypes; covering system-prompt extraction, instruction-override, role-manipulation, multi-turn injection, and indirect injection via user-provided content; run on every deployment update for Critical/High-tier; failure blocks go-live.
- **Jailbreak corpus**, 30–100 jailbreak inputs targeting role-override, persona-switch, authority-claim, encoding-bypass, and multi-step reasoning chains; run against chatbot and AI assistant archetypes on every deployment update.
- **Multi-modal injection corpus**, 20–60 image and audio inputs embedding encoded prompt-injection instructions (steganographic payloads, supersonic embedding, adversarial overlays); run against multi-modal archetypes on every deployment update.
- **DLP-paste-block corpus**, 20–50 regulated-data canary inputs (credit card patterns, SSN-format strings, synthetic PHI, synthetic source-code fragments) used to verify DLP enforcement is active at the endpoint boundary; run on periodic cadence and on any DLP policy change.
- **Browser-extension-scope corpus**, 20–50 crafted page configurations testing that browser-based AI tools cannot read or modify pages outside their declared host-permission scope; run on every extension version update.
- **Art. 50 disclosure corpus**, 10–20 opening-interaction test fixtures verifying that the AI disclosure is present at the first interaction and at defined interaction points for each customer-facing chatbot and multi-modal interface; run on every UX release.

Corpus management:
- Versioned in source control; corpus changes go through review with a named corpus owner.
- Corpus refresh cadence: monthly minimum from three sources, internal observations (IR findings, IM incidents, red-team results), external public corpora (OWASP LLM Top 10 examples, ATLAS technique examples, OWASP MASVS test cases, public jailbreak research), and sector-specific advisory content.
- New deployment adding or modifying AI/HAI endpoint capabilities triggers a corpus completeness check before go-live.

**C) Operate the go-deployment battery and wire test failures to IM**

Every AI/HAI-enabled endpoint must pass its archetype battery before receiving Sanctioned status in the SM-Endpoints inventory. Go-deployment triggers:
- **Pre-production:** all applicable archetype tests must pass before the endpoint capability is promoted to production; the go-deployment test record is linked from the SM-Endpoints inventory.
- **Post-model-update:** any model version change, firmware update, or app version change triggers a re-run of the applicable archetype battery within 14 days (Critical-tier: within 7 days).
- **Post-incident:** any IM-Endpoints incident involving the endpoint triggers a re-run of the relevant battery subset before the incident is closed.
- **Quarterly:** all active AI/HAI-enabled endpoints re-run their battery; results reviewed by the named test-battery owner.

All test failures route to IM-Endpoints within one business day with a severity tag. Named battery owner per archetype; battery ownership is a named role, not a shared-team responsibility.

#### Outcome Metrics (L1)

| Metric | Baseline | L1 Target | Source |
|---|---|---|---|
| % of AI/HAI-enabled endpoints reaching production with a passed go-deployment battery on record | measure | ≥90% within 12 months; 100% for Critical/High-tier | SM-Endpoints inventory × test-run registry |
| Regression corpora published (prompt-injection, jailbreak, multi-modal injection, DLP-paste-block, browser-extension-scope, Art. 50 disclosure) | 0 / 6 | 6 / 6 | Corpus registry |
| % Art. 50 disclosure-presence tests passing on every deployment update for customer-facing chatbots | measure | 100% | Test-run registry |
| % archetype threat library entries covered by at least one test or corpus entry | measure | ≥80% by end of year 1 | TA library × test metadata |
| % test failures routed to IM within 1 business day | measure | 100% | Test → IM handoff metrics |

#### Process Metrics (leading)

- Battery owner named per archetype; quarterly re-run scheduled in advance.
- Test-automation coverage, target ≥60% of battery items running without human intervention at L1.
- Corpus refresh cadence honored monthly; corpus owner reviews incoming sources.
- New-archetype lead time, from "first deployment of a new endpoint archetype" to "battery published" ≤30 days.

#### Effectiveness Metrics (business value)

- Pre-production catch rate, test failures that uncovered a misconfiguration or control gap before production deployment (prompt injection in a new chatbot, Art. 50 disclosure absent from a new multi-modal interface, DLP-paste-block not wired in an AI assistant rollout).
- Post-incident corpus growth, red-team findings and IM incidents converted to corpus entries that catch the same failure pattern before the next deployment.
- Reduced IM incident volume, endpoints with a full battery pass-rate in CI/CD have a measurably lower IM incident rate than those without.

#### Success Criteria

- Per-archetype foundational test battery published for all seven archetypes (AI assistant on managed endpoint, browser-based AI tool, chatbot / conversational UI, multi-modal AI interface, SaaS-AI productivity feature, mobile AI app, edge AI device), linked from the SM-Endpoints inventory record and the DR/IR artifacts.
- Six regression corpora published in source control (prompt-injection, jailbreak, multi-modal injection, DLP-paste-block, browser-extension-scope, Art. 50 disclosure), with a named corpus owner and a monthly refresh cadence.
- 100% of AI/HAI-enabled endpoints reaching production in the last 90 days have a passed go-deployment battery on record.
- All test failures routed to IM with a 1-day handoff SLA and named owner.
- Named battery owner per archetype; automation covers ≥60% of battery items.

---

## Maturity Level 2
### Objective: Calibrate test depth per risk tier using the SM-Endpoints L2 tier-treatment matrix, run scheduled per-tier red-team exercises for Critical (quarterly) and High (semi-annual) endpoints, and test cross-archetype compositions for Critical-tier endpoints

At this level, testing stops treating all AI/HAI-enabled endpoints the same. Per-tier calibration drives what each endpoint receives from the test program. Critical-tier endpoints are red-teamed quarterly; High-tier semi-annually. TA-Endpoints L2 deep threat models replace archetype snapshots as the scenario library for red-team exercises. Cross-archetype compositions (e.g., a managed-endpoint AI assistant that also uses a browser extension; a customer-facing chatbot with a multi-modal interface) receive combined test suites that exercise composition-specific failure modes.

#### Dependencies

- **ST-Endpoints L1 (required):** per-archetype batteries and regression corpora must be operational before per-tier calibration is meaningful.
- **SM-Endpoints L2 (required):** the risk-tier rubric (Critical / High / Medium / Low) and tier-treatment matrix determine which endpoints receive full-scope testing vs. a subset.
- **TA-Endpoints L2 (required):** per-artifact deep threat models for Critical-tier endpoints shape red-team scope; archetype-only snapshots are insufficient for L2 red-team exercises.
- **Supports / unblocks:** IM-Endpoints L2 (tier-calibrated incident handling relies on tier-calibrated test signals), ML-Endpoints L2 (detections tuned per-artifact threat model depend on L2 test findings to validate coverage).

#### Desired Outcomes

- Test depth is visibly differentiated: Critical-tier endpoints get the full battery plus quarterly red-team and cross-archetype composition tests; Low-tier get the base corpora and a spot-check.
- Red-team findings are scenario-based (from TA-Endpoints L2 per-artifact deep threat models), not free-form; findings trace to specific ATLAS tactics and SR requirements.
- Regression corpora grow from red-team findings: every Critical or High-severity red-team finding produces a new corpus entry within 30 days.
- Cross-archetype composition failure modes (AI assistant + browser extension interaction; chatbot + multi-modal composition) are in scope and have documented test coverage.
- Per-tier SLA adherence for testing activities is tracked and reported to the program sponsor.

#### Activities

**A) Tier-calibrated test battery and corpus depth**

Publish a per-tier test treatment aligned to SM-Endpoints L2's tier-treatment matrix:

| Treatment | Critical | High | Medium | Low |
|---|---|---|---|---|
| Go-deployment battery | Full archetype battery, all test classes, executive sign-off on results | Full archetype battery | Subset battery (top-4 threat classes) | Spot-check (3 test classes) |
| Regression corpus | All 6 corpora running on every deployment update | All 6 corpora on deployment + periodic | Prompt-injection + jailbreak corpora on deployment | Prompt-injection corpus on deployment |
| Post-model-update re-run | Full battery within 7 days | Full battery within 14 days | Subset battery within 30 days | Output-integrity check at next quarterly |
| Art. 50 disclosure test | Automated probe on every UX release; failure is P1 | Automated probe on every UX release | Verified at go-deployment and annually | Verified at go-deployment |
| DLP-paste-block test | Verified quarterly by ST; findings route to IM within 1 BD | Verified semi-annually | Verified annually | Verified at go-deployment |

**B) Scheduled per-tier red-team exercises using TA-Endpoints L2 threat models**

Red-team cadence by tier:
- **Critical:** quarterly (4 per year); scope derived from TA-Endpoints L2 per-artifact deep threat model; covers prompt-injection chains, indirect prompt injection via user-provided content, DLP bypass attempts, Art. 50 disclosure circumvention, tool-allowlist escape (AI assistant), extension-scope violation (browser tool), data-exfiltration via AI completions, physical-interface attacks (edge device), biometric-bypass via synthetic media (mobile), silent SaaS feature enablement.
- **High:** semi-annual (2 per year); scope from TA-Endpoints L2 artifact deltas; covers the top-5 threats from the per-artifact model.
- **Medium/Low:** ad-hoc (before major configuration changes or scope expansions); archetype snapshot drives scope.

Each red-team exercise follows the HAIAMM ST methodology: written rules of engagement, test plan reviewed with the endpoint owner, execution log, structured findings report (severity / root cause / ATLAS tactic ID / SR requirement traced), remediation pairings. Red-team findings at Critical or High severity produce corpus entries within 30 days.

Cross-archetype composition tests for Critical-tier:
- **AI assistant + browser extension composition:** browser extension passes content to the AI assistant that contains injected instructions; verify the assistant does not follow the injected instructions; verify the DLP policy covers the combined data-flow path.
- **Chatbot + multi-modal composition:** submit a multi-modal input (image + text) to a chatbot interface; verify the cross-modal safety filter applies consistently; verify Art. 50 disclosure is present regardless of input modality; test for indirect prompt injection via image content flowing into the text-response generation path.

**C) Red-team findings → corpus pipeline**

Every Critical or High-severity red-team finding produces:
1. A new corpus entry (input, expected safe output, threat tag, ATLAS tactic ID, date, source reference) committed to the relevant regression corpus within 30 days.
2. An IM-Endpoints finding with severity tag and the named endpoint owner as assignee.
3. A TA-Endpoints library-gap ticket if the finding was not in the archetype library, tracked with a named owner and a 30-day close SLA for Critical-tier gaps.

#### Outcome Metrics (L2)

| Metric | Baseline | L2 Target | Source |
|---|---|---|---|
| % Critical-tier endpoints red-teamed in last 90 days | measure | 100% | ST records |
| % High-tier endpoints red-teamed in last 180 days | measure | 100% | ST records |
| Regression corpus growth rate, Critical-tier corpora | measure | ≥1 new entry per month from red-team or incident findings | Corpus change-log |
| % red-team findings (Critical/High severity) converted to corpus entries within 30 days | measure | ≥90% | Finding → corpus pipeline telemetry |
| Per-tier SLA adherence for testing activities | measure | ≥90% per tier | Program telemetry |

#### Process Metrics (leading)

- Red-team schedule on calendar; no Critical-tier endpoint skips a quarterly exercise.
- Corpus review cadence, monthly.
- Cross-archetype composition test plan published for each Critical-tier endpoint with a composition; reviewed by named architect.
- Finding → TA library-gap pipeline: Critical gaps closed within 30 days; High within 60 days.

#### Effectiveness Metrics (business value)

- Pre-production catch rate for Critical-tier increases as red-team exercises catch issues before configuration updates deploy.
- Corpus catches post-update regressions early, failures that would have reached production users, detected by corpus before go-live.
- Incident rate for Critical-tier endpoints with full-scope testing is measurably lower than for those without.

#### Success Criteria

- Quarterly red-team for 100% of Critical-tier endpoints; semi-annual for 100% of High-tier; scope tied to TA-Endpoints L2 per-artifact deep threat models.
- Critical-tier regression corpora (all 6) running on every deployment update for all Critical-tier endpoints; per-tier calibration enforced.
- ≥90% of Critical/High-severity red-team findings converted to corpus entries within 30 days.
- Cross-archetype composition tests documented and run for all Critical-tier endpoints with composite archetypes.
- Per-tier SLA adherence for testing activities ≥90%.

---

## Maturity Level 3
### Objective: Operate continuous automated adversarial testing for Critical-tier endpoints, publish regression corpora and findings as open artifacts, and contribute discovered TTPs to MITRE ATLAS, AVID, OWASP MASVS, OWASP Browser-Extension Top 10, and CSA endpoint

At this level, testing runs continuously rather than periodically. An automated adversarial testing harness probes Critical-tier endpoints daily using generated prompt-injection ladders, multi-modal injection seeders, DLP-bypass generators, and extension-scope probes. Novel findings are triaged into the TA-Endpoints library weekly. Anonymized regression corpora, test patterns, and discovered TTPs are contributed to MITRE ATLAS, AVID, OWASP MASVS, OWASP Browser-Extension Top 10, and CSA endpoint working groups.

#### Dependencies

- **ST-Endpoints L2 (required):** per-tier red-team function, regression corpora, and finding → corpus pipeline must be operational.
- **TA-Endpoints L3 (required):** automated TTP-ingestion pipeline from ST telemetry feeds the TA L3 auto-update loop.
- **ML-Endpoints L2+ (required):** endpoint monitoring instrumented to catch automated-probe activity so the harness and the monitoring pipeline are calibrated against each other.
- **SM-Endpoints L3 (alignment):** automated inventory signals feed the automated test harness with the current Critical-tier endpoint list and their tier metadata.

#### Desired Outcomes

- Critical-tier adversarial posture is measured daily, not quarterly.
- Novel TTPs discovered by the automated harness reach the TA-Endpoints library within 14 days.
- Program-originated findings appear in MITRE ATLAS, AVID, OWASP MASVS, OWASP Browser-Extension Top 10, and CSA endpoint revisions.
- Published regression corpora and test patterns are adopted by peer organizations, demonstrating industry-leading endpoint AI testing maturity.

#### Activities

**A) Continuous automated adversarial testing harness**

Deploy an automated adversarial testing harness that runs daily against all Critical-tier AI/HAI-enabled endpoints:

- **Prompt-injection generator:** produces novel direct and indirect prompt-injection inputs using mutation of the regression corpus, template-based variation, and generated jailbreak-ladder sequences; runs against chatbot, AI assistant, and multi-modal archetypes; confirms output filters and prompt-injection defenses are active.
- **Multi-modal injection seeder:** generates image and audio inputs embedding adversarial instructions (steganographic payloads, supersonic audio embedding, adversarial overlay generation); exercises the multi-modal input validation and safety-filter paths.
- **DLP-bypass generator:** generates regulated-data canary variants (encoding variations, tokenization-splitting attempts, homoglyph substitution) designed to evade DLP pattern-match rules; verifies DLP enforcement is robust against common evasion patterns.
- **Extension-scope probe:** generates crafted page configurations and cross-origin access attempts that exercise browser extension host-permission enforcement; verifies extensions cannot read or modify out-of-scope pages.
- **Art. 50 disclosure monitor:** automated daily probe of all live customer-facing chatbot and multi-modal interfaces confirming AI disclosure is present at the first interaction; any interface where disclosure is absent generates a P1 finding to IM-Endpoints within 1 hour.

Findings triaged by a named ST owner at least weekly. New TTPs (patterns not in the TA-Endpoints library) fed into the TA L3 auto-proposal pipeline within 14 days. High-severity automated findings route to IM-Endpoints within 24 hours.

**B) Contribute findings to industry**

Contribute anonymized, legally-vetted findings to:
- **MITRE ATLAS**, novel prompt-injection technique observations for endpoint-specific surfaces (voice injection, image embedding, browser-extension injection chains); submissions follow ATLAS evidence-and-provenance requirements; target ≥4 contributions per year.
- **AI Vulnerability Database (AVID)**, structured disclosure submissions for novel vulnerabilities in AI/HAI endpoint deployments or their upstream AI vendors (coordinated disclosure where third-party components are involved).
- **OWASP MASVS**, real-world test evidence for mobile AI app security requirements; target ≥2 substantive submissions per revision cycle.
- **OWASP Browser-Extension Top 10**, real-world telemetry evidence for browser-based AI tool attack surface during revision cycles.
- **CSA endpoint working groups**, anonymized findings and test patterns for managed-endpoint and SaaS-AI feature attack surfaces; participation in joint red-team benchmarking exercises.

**C) Publish regression corpora and test patterns as open artifacts**

- Publish anonymized versions of all six regression corpora (prompt-injection, jailbreak, multi-modal injection, DLP-paste-block, browser-extension-scope, Art. 50 disclosure) under an open license; scrubbed of org-specific endpoint names, data classes, and identifiers.
- Maintain the published versions upstream; internal corpora are a superset of the published versions with org-specific entries not shared externally.
- Host or co-host at least one industry endpoint AI red-team benchmark per year (OWASP AI chapter, CSA endpoint working group, sector ISAC AI red-team exercise); collect cross-org detection-benchmark improvement data.

#### Outcome Metrics (L3)

| Metric | Baseline | L3 Target | Source |
|---|---|---|---|
| % Critical-tier endpoints under continuous automated adversarial testing (daily probe execution) | measure | ≥80% | ST harness telemetry |
| New-TTP ingestion lead time (automated finding to TA-Endpoints library entry) | measure | ≤14 days | Harness → TA pipeline telemetry |
| Industry contributions per year (MITRE ATLAS / AVID / OWASP MASVS / Browser-Extension Top 10 / CSA) | 0 | ≥4 | Contribution log |
| Open regression corpora published and maintained upstream | 0 | ≥6 corpora published | External repository |
| Art. 50 disclosure probe, % of live customer-facing chatbot and multi-modal interfaces passing daily disclosure check | measure | 100% | Automated probe telemetry |

#### Process Metrics (leading)

- Continuous harness health, % Critical-tier endpoints producing a fresh automated probe result within the last 24 hours; on-call paged when a harness feed goes stale >24 hours.
- New-TTP triage cadence, automated probe findings reviewed weekly by named ST owner; novel patterns forwarded to TA-Endpoints library within 14 days.
- Art. 50 disclosure probe health, all in-scope customer-facing interfaces covered by the daily disclosure probe; any gap treated as a P1 process failure.
- Industry-contribution pipeline, at least one anonymized finding in-preparation, in-legal-review, or submitted at any time.

#### Effectiveness Metrics (business value)

- Mean time to detect novel endpoint AI attack techniques decreases as the continuous harness catches model-update regressions within hours, not sprint cycles.
- Program-originated TTPs recognized in MITRE ATLAS, AVID, OWASP MASVS, or OWASP Browser-Extension Top 10 demonstrate external validation of testing rigor.
- Zero incidents attributable to an Art. 50 disclosure being absent from a customer-facing interface, the daily disclosure probe closes the gap between quarterly audits and continuous compliance.
- Cross-org exercise participants cite improved detection benchmarks from shared corpora.

#### Success Criteria

- ≥80% of Critical-tier AI/HAI-enabled endpoints under continuous automated adversarial testing with daily probe execution.
- New-TTP ingestion lead time ≤14 days; automated findings triaged weekly by named owner.
- ≥4 industry contributions per year to MITRE ATLAS / AVID / OWASP MASVS / OWASP Browser-Extension Top 10 / CSA endpoint.
- All six open regression corpora published under a permissive license and maintained upstream.
- Art. 50 disclosure daily probe covering 100% of in-scope customer-facing interfaces; ≥1 hosted industry endpoint AI red-team benchmark per year plus ≥2 participated.

---

## Key Success Indicators

**Level 1:**
- Per-archetype foundational test battery published for all seven archetypes (AI assistant on managed endpoint, browser-based AI tool, chatbot / conversational UI, multi-modal AI interface, SaaS-AI productivity feature, mobile AI app, edge AI device), each test class tied to a TA-Endpoints archetype threat and an SR-Endpoints requirement.
- Six regression corpora (prompt-injection, jailbreak, multi-modal injection, DLP-paste-block, browser-extension-scope, Art. 50 disclosure) versioned in source control, with named corpus owner and monthly refresh cadence.
- 100% of AI/HAI-enabled endpoints reaching production in the last 90 days have a passed go-deployment battery on record linked from the SM-Endpoints inventory.
- All test failures routed to IM within 1 business day; automation covers ≥60% of battery items.
- TA-Endpoints archetype threat coverage by test battery ≥80%.

**Level 2:**
- 100% of Critical-tier endpoints red-teamed quarterly; 100% of High-tier semi-annually; scope tied to TA-Endpoints L2 per-artifact deep threat models.
- Per-tier calibration enforced: Critical-tier gets all 6 corpora on every deployment update; Low-tier gets prompt-injection corpus on deployment.
- ≥90% of Critical/High-severity red-team findings converted to corpus entries within 30 days.
- Cross-archetype composition tests (AI assistant + browser extension; chatbot + multi-modal) documented and run for all Critical-tier composite endpoints.

**Level 3:**
- ≥80% of Critical-tier endpoints under continuous automated adversarial testing with daily probe execution; novel TTPs reaching TA-Endpoints library within 14 days.
- ≥4 industry contributions per year to MITRE ATLAS / AVID / OWASP MASVS / OWASP Browser-Extension Top 10 / CSA endpoint; all six open regression corpora published and maintained.
- Art. 50 disclosure daily probe passing for 100% of in-scope customer-facing interfaces; ≥1 hosted + ≥2 participated endpoint AI red-team exercises per year.

---

## Common Pitfalls

**Level 1:**
- ❌ Test battery reduced to a logging-completeness check and a functional happy-path assertion, no adversarial probes (prompt injection, DLP-paste-block, Art. 50 disclosure check, extension-scope test) actually exercised.
- ❌ Regression corpora committed to source control but not wired into the deployment pipeline, they exist but run only when a reviewer manually triggers them; coverage erodes after every update cycle.
- ❌ Art. 50 disclosure test runs once at go-deployment but is never re-run after UX releases, a UX refresh removes the disclosure; the test is never re-executed; the absence goes undetected until a regulator or user reports it.
- ❌ Go-deployment battery runs once pre-production but is never re-run after model-version updates, firmware updates, or app version changes.
- ❌ Test failures logged in a spreadsheet separate from IM, no SLA enforcement, no aging visibility, no named owner; the same failure recurs across multiple deployments undetected.
- ❌ Corpus refresh skipped for three months because "nothing new happened", public jailbreak research, OWASP MASVS updates, and ATLAS technique additions pile up unread; the corpus is frozen at launch while the external threat landscape advances.
- ❌ Test battery conflates chatbot and AI assistant, chatbot-specific tests (Art. 50 disclosure, escalation path, prompt-injection corpus) are absent because the same checklist was applied to both archetypes.

**Level 2:**
- ❌ Red-team scope defined as "prompt-injection probes" but indirect injection via user-provided image content, DLP bypass attempts, Art. 50 disclosure circumvention, and cross-archetype composition failures are excluded, the top threat classes for chatbot + multi-modal compositions go untested.
- ❌ Corpus growth declared at ≥1 per month but entries are variations on the same payload format, the corpus does not cover new jailbreak techniques published since L2 launch; model-update regressions pass the corpus and fail in production.
- ❌ Per-tier calibration documented in the tier-treatment matrix but the deployment pipeline applies the same corpus to all tiers, Critical endpoints run the same tests as Low; differentiation exists on paper only.
- ❌ Red-team findings route to IM but the finding → corpus pipeline is never executed, 12 months of Critical/High findings sit in IM as closed tickets with no corpus entries; the same vulnerabilities are re-discovered at the next red-team exercise.
- ❌ Cross-archetype composition tests scoped but not executed because "no team owns AI-assistant-plus-browser-extension testing", composition-specific failure modes are in the threat model but not in any test.

**Level 3:**
- ❌ Continuous harness runs prompt-injection probes that the chatbot's output filter trivially blocks, coverage metric looks good but the probes are not exercising real novel attack surfaces; new jailbreak techniques and multi-modal injection variants are not generated.
- ❌ Art. 50 disclosure daily probe covers only the initial interaction of the primary chatbot, SaaS-AI productivity features and multi-modal interfaces on secondary surfaces are not probed; a disclosure gap persists undetected.
- ❌ Industry contributions are legal-vetted case-study summaries rather than actionable, reproducible technique descriptions, ATLAS reviewers cannot map them to a technique ID; OWASP MASVS submissions lack test-case reproducibility notes.
- ❌ Open corpora published once and then not maintained, external organizations build on a stale version while the internal corpus has 60 new entries; discrepancies surface at community exercises and damage the program's credibility.
- ❌ New-TTP ingestion from automated probes to the TA-Endpoints library is manual and quarterly, by the time a novel endpoint AI technique reaches SR and SA updates, the technique is already being exploited in the wild.

---

## Practice Maturity Questions

**Level 1:**
1. Is a per-archetype foundational test battery published for all seven AI/HAI endpoint archetypes, with each test class tied to a TA-Endpoints archetype threat (HAI TTP + ATLAS tactic ID) and an SR-Endpoints requirement, defined inputs/outputs/pass-fail criteria, and an evidence artifact, and are 100% of new AI/HAI-enabled endpoints required to pass the battery before production Sanctioned status is issued?
2. Are six regression corpora (prompt-injection, jailbreak, multi-modal injection, DLP-paste-block, browser-extension-scope, Art. 50 disclosure) versioned in source control, with a named corpus owner, a monthly refresh cadence from internal and external sources, and runs triggered on deployment updates for Critical/High-tier endpoints, and is the Art. 50 disclosure corpus re-run on every UX release for customer-facing chatbots and multi-modal interfaces?
3. Are all test failures routed to IM-Endpoints within 1 business day with a severity tag and named owner, and does TA-Endpoints archetype threat coverage by the test battery and corpus reach ≥80% by end of year one?

**Level 2:**
1. Are 100% of Critical-tier AI/HAI-enabled endpoints red-teamed at least quarterly, and 100% of High-tier semi-annually, with scope derived from TA-Endpoints L2 per-artifact deep threat models, covering prompt-injection chains, multi-modal injection, DLP bypass, Art. 50 disclosure circumvention, tool-allowlist escape, extension-scope violation, data-exfiltration probes, and physical-interface attacks for applicable archetypes, with findings routed to IM and remediation tracked?
2. Is per-tier corpus calibration enforced (Critical-tier: all 6 corpora on every deployment update; Low-tier: prompt-injection corpus on deployment), and are ≥90% of Critical/High-severity red-team findings converted to corpus entries within 30 days?
3. Are cross-archetype composition tests (AI assistant + browser extension; chatbot + multi-modal) documented and executed for all Critical-tier composite endpoints, and is per-tier SLA adherence for testing activities ≥90%?

**Level 3:**
1. Are ≥80% of Critical-tier AI/HAI-enabled endpoints under continuous automated adversarial testing with daily probe execution, covering prompt injection, multi-modal injection, DLP-bypass generation, extension-scope probing, and Art. 50 disclosure monitoring, with novel TTPs triaged into the TA-Endpoints library within 14 days and high-severity automated findings routed to IM within 24 hours?
2. Has the program contributed ≥4 anonymized, legally-vetted findings per year to MITRE ATLAS, AVID, OWASP MASVS, or OWASP Browser-Extension Top 10, and are all six open regression corpora published under a permissive license and maintained upstream with documented external adoption?
3. Is the Art. 50 disclosure daily probe covering 100% of in-scope customer-facing chatbot and multi-modal interfaces, and has the program hosted at least 1 industry endpoint AI red-team benchmark per year and participated in ≥2 additional cross-org exercises, with documented cross-org detection-benchmark improvement data?

---

**Document Version:** HAIAMM v3.0
**Practice:** Security Testing (ST)
**Domain:** Endpoints
**Last Updated:** 2026-05-14
**Author:** Verifhai