Issue Management (IM)

Infrastructure Domain - HAIAMM v3.0

Practice Overview

Objective: Run the single unified backlog for all AI infrastructure issues, findings from TA-Infrastructure threat snapshots, SR-Infrastructure gaps, DR-Infrastructure conditions, IR-Infrastructure drift, ST-Infrastructure failures, ML-Infrastructure detections, and external advisories (CVEs for AI-infra components, cloud-provider security bulletins, CNCF advisories, ATLAS updates), with a tier-calibrated incident playbook containing AI-infrastructure-specific containment plays, and regulatory SLA tracking covering GDPR Art. 33, EU AI Act Art. 73, HIPAA, NYDFS Part 500, PCI-DSS, and sector cloud regulations (FedRAMP IR, ISO/IEC 27035).

Description: IM-Infrastructure is the clearinghouse for everything the other Infrastructure-domain practices produce. Every TA-Infrastructure threat snapshot row carrying residual risk, every SR-Infrastructure REM accepted gap with an owner and expiry, every DR-Infrastructure approve-with-conditions item, every IR-Infrastructure drift finding, every ST-Infrastructure test failure, every ML-Infrastructure detection that fires, and every external advisory (CVE for AI-infra components, cloud-provider security bulletins, CNCF security advisories, ATLAS updates for AI infrastructure TTPs) flows into a single prioritized backlog with named owners, tier-calibrated SLAs, and an unambiguous incident playbook. The playbook contains AI-infrastructure-specific containment plays: cross-tenant breach containment, model registry compromise containment, GPU residual-state leakage containment, orchestrator compromise containment, vector-store mass-extraction containment, AI CI/CD pipeline compromise containment, and shadow inference endpoint containment. Every Critical or blocker incident receives a post-incident review whose outputs flow back to SA-Infrastructure, SR-Infrastructure, EG-Infrastructure, and ML-Infrastructure. The regulatory SLA tracker ensures GDPR Art. 33 72-hour breach notification, EU AI Act Art. 73 serious-incident reporting, FedRAMP IR, and other applicable sector cloud notification windows are never missed because of organizational diffusion.

Context: Without a unified backlog, AI infrastructure issues scatter across platform Jira projects, cloud-security queues, SRE incident channels, and ML-platform alert dashboards. TA-Infrastructure residual risks age without remediation owners. IR-Infrastructure drift findings sit in a spreadsheet that nobody updates between annual reviews. An ML-Infrastructure GPU clearing failure fires on a Friday and routes to the on-call SRE who does not recognize it as a security-relevant event. An orchestrator control plane is compromised and the first signal is customer-visible workflow misbehavior two days later. The GDPR Art. 33 72-hour clock starts at the moment the organization becomes aware of a personal data breach, not when the responsible team finishes triaging the inference-endpoint request logs. IM-Infrastructure closes all of these gaps with a single backlog, a single triage rubric, and a named on-call path for every severity class.

Maturity Level 1

Objective: Operate a single unified AI infrastructure issue backlog with a standard triage rubric, AI-infrastructure-specific incident playbook with containment plays for the primary AI infrastructure incident classes, and regulatory SLA tracking for GDPR Art. 33, EU AI Act Art. 73, HIPAA, NYDFS Part 500, PCI-DSS, FedRAMP IR, and ISO/IEC 27035

At this level, every AI infrastructure issue has a home, a severity, an owner, and an SLA, and infrastructure incidents follow a named playbook with AI-specific containment actions rather than generic cloud-incident response.

Dependencies

• SM-Infrastructure L1 (required): the inventory provides the affected-component and owning-team spine for every issue; without the inventory, the backlog cannot route issues to accountable owners.
• PC-Infrastructure L1 (required): priority compliance map anchors the regulatory SLA tracker; triage rubric severity definitions reference compliance exposure from GDPR Art. 33, EU AI Act Art. 73, and sector cloud regulations.
• TA-Infrastructure L1 (required): archetype threat library drives the incident classification taxonomy and the AI-specific incident classes in the playbook.
• ML-Infrastructure L1 (required): detections from ML-Infrastructure are the primary runtime input to the issue backlog; without ML L1, the detection-to-backlog feed does not exist and runtime infrastructure incidents go unreported until externally visible.
• Supports / unblocks: ML-Infrastructure L1 (post-incident reviews feed detection-tuning back into ML); SA-Infrastructure L1 (post-incident reviews generate pattern-update requests); SR-Infrastructure L1 (post-incident reviews generate requirements-pack update requests); EG-Infrastructure L1 (incident trends feed training-content updates).

Desired Outcomes

• One backlog, one triage rubric, one incident playbook for all AI infrastructure issues, regardless of source practice.
• AI-infrastructure-specific incident classes are handled on named playbook entries with pre-assigned roles, containment steps, and SLA targets, not improvised at incident time.
• Regulatory SLA tracker is live and showing zero missed notification windows for GDPR Art. 33, EU AI Act Art. 73, HIPAA, NYDFS Part 500, PCI-DSS, FedRAMP IR, and ISO/IEC 27035 in the last 90 days.
• Post-incident reviews for Critical / blocker incidents produce named updates to SA-Infrastructure, SR-Infrastructure, EG-Infrastructure, and ML-Infrastructure within 14 days of incident closure.
• Backlog aging is visible to the program sponsor monthly; a small number of aging buckets are actively managed with clear escalation paths.

Activities

A) Stand up the AI infrastructure issue backlog and triage rubric

One backlog with standardized metadata:

• Source, TA (threat snapshot residual risk) / SR (REM accepted gap) / DR (approve-with-conditions item) / IR (drift finding) / ST (test failure) / ML (detection alert) / External (CVE for AI-infra component, cloud-provider security bulletin, CNCF security advisory, ATLAS update, sector ISAC advisory).
• Affected component(s), linked to SM-Infrastructure inventory; archetype, tier, owning team.
• Severity, Critical / High / Medium / Low per the rubric below.
• Owner, named component owner from the SM-Infrastructure inventory; escalation path to program sponsor.
• SLA, severity-based (published at L1; per-tier calibrated at L2).
• Evidence, link to originating artifact (TA snapshot row, REM gap row, DR decision, IR finding, ST test result, ML alert ticket, external advisory URL).
• Regulatory flag, whether the issue carries a regulatory notification obligation.

Severity rubric (AI infrastructure specific):

• Critical, active cross-tenant data access through an inference endpoint or vector store; confirmed GPU residual-state leakage between tenants; orchestrator control-plane compromise with confirmed workflow execution under attacker control; model registry compromise with unsigned artifact promoted to production; personal data breach through AI infrastructure processing that triggers GDPR Art. 33; regulated data confirmed in transit through a shadow inference endpoint.
• High, confirmed control failure in a production AI infrastructure component with potential for harm if not contained (unsigned model artifact in production without confirmed breach; vector-store query bypassing retrieval-policy without confirmed exfiltration; GPU clearing failure with single-tenant impact; shadow inference endpoint detected with no confirmed data transit of regulated content).
• Medium, confirmed gap in a non-production component or a production component with compensating controls active; no current active impact; SR REM accepted-gap past expiry with no renewal; IR drift finding on a Medium-tier component; base-image critical CVE not yet patched within SLA.
• Low, informational; non-urgent gap; external advisory not yet assessed; Low-tier component logging baseline gap; SBOM missing for a non-Critical component.

SLA targets (published at L1, per-tier calibrated at L2 per SM-Infrastructure L2 matrix):

• Critical: acknowledge ≤4 hours / contain ≤48 hours / root-cause ≤30 days.
• High: acknowledge ≤24 hours / contain ≤7 days / root-cause ≤45 days.
• Medium: acknowledge ≤48 hours / remediate ≤14 days.
• Low: acknowledge ≤5 business days / remediate ≤30 days.

Triage cadence: daily review for Critical and new High; weekly queue review for Medium; monthly aging review for the full backlog.

B) Publish the AI-infrastructure-specific incident playbook

Publish playbook entries for the primary AI infrastructure incident classes. Each entry includes: trigger conditions, named roles (infrastructure deployer-duty owner, cloud-security on-call, Privacy/Legal contact, executive sponsor escalation path), step-by-step containment, artifacts to collect, evidence-capture instructions, closure criteria, and SLA targets.

Cross-tenant breach containment: Trigger: ML-Infrastructure cross-tenant access detection (principal tenant-id does not match retrieved-resource tenant-id on inference endpoint or vector store). Containment: (1) disable the affected inference endpoint or vector-store query path via feature flag or network policy; (2) assess scope, which tenants, which sessions, what data was accessed; (3) perform lineage audit of all affected tenants to determine if any personal data was accessible; (4) evaluate GDPR Art. 33 trigger, if personal data of an EU data subject was accessed without authorization, the 72-hour clock starts at first internal awareness; (5) route to Privacy/Legal immediately; (6) rotate access credentials for the affected multi-tenant component; (7) apply infrastructure-layer per-tenant isolation remediation if absent. Evidence: ML alert record, request-event log export for the affected period, per-tenant retrieval log, GDPR Art. 33 evaluation record.

Model registry compromise containment: Trigger: ML-Infrastructure unsigned model artifact detected in production, or promotion event without approver, or external advisory indicating model registry supply-chain incident (ATLAS AML.T0010). Containment: (1) freeze all model promotions, set registry promotion gate to manual-approval-only mode; (2) rollback to the last known-good model version in all affected inference endpoints; (3) audit all promotions in the past 30 days, compare against the approved-promotion log; identify any unsigned or unapproved promotions; (4) rotate registry service-account credentials and model-signing keys; (5) re-verify SLSA provenance attestations for all production model artifacts; (6) assess whether compromised model artifacts processed regulated data and evaluate GDPR Art. 33; (7) file a supply-chain incident report referencing ATLAS AML.T0010 with the appropriate ISAC. Evidence: promotion-event log export, signature-verification event log, rollback event record, SLSA attestation verification results.

GPU residual-state leakage containment: Trigger: ML-Infrastructure GPU residual-state clearing failure event, or ST-Infrastructure residual-state test identifying cross-tenant tensor exposure. Containment: (1) drain affected nodes immediately, cordon nodes and evict all running jobs; (2) audit the clearing-process configuration, identify the root cause of the clearing failure (missing DaemonSet, driver version regression, configuration drift); (3) assess which workloads ran on the affected nodes and in what order, determine if any higher-classification workload's residual state was accessible to a lower-classification workload; (4) if personal data or regulated data may have been exposed, evaluate GDPR Art. 33 and applicable sector cloud regulation notification obligations; (5) remediate the clearing configuration; (6) verify clearing succeeds on all nodes before returning to production. Evidence: clearing-event log showing failure, node-scheduling event log for the affected period, workload classification records, GDPR Art. 33 evaluation record.

Orchestrator compromise containment: Trigger: ML-Infrastructure orchestrator workflow injection detection (step-principal mismatch), or ST-Infrastructure workflow-integrity test failure, or external advisory for the orchestrator platform. Containment: (1) kill all active workflows, terminate all running orchestrator sessions; (2) rotate orchestrator service-account credentials and API tokens; (3) audit agent-state events for the affected time window, identify any unauthorized actions taken by injected workflow steps; (4) assess downstream effects, if a compromised workflow step made writes to external systems (databases, APIs, file stores), assess whether the writes must be rolled back; (5) review the workflow definition reference for tampered step definitions; (6) if any personal data or regulated data was accessed or exfiltrated by the injected workflow, evaluate GDPR Art. 33 and EU AI Act Art. 73. Evidence: workflow-execution event log, step-principal event log, agent-state event log, downstream-system write audit.

Vector-store mass-extraction containment: Trigger: ML-Infrastructure vector-store extraction pattern detection (anomalous retrieval volume for a principal), or DLP alert on bulk embedding export. Containment: (1) disable the vector-store query path for the affected principal or tenant, apply a retrieval-policy block; (2) assess scope, which corpus, which tenant's data, what document IDs were retrieved; (3) apply a classification-gated query allowlist that restricts retrieval to approved document classes until the investigation is complete; (4) evaluate GDPR Art. 33 if the extracted embeddings can be correlated back to personal data (embedding inversion risk assessment); (5) assess whether the exfiltration destination is an external party and whether applicable data breach notification is triggered; (6) notify affected tenants if per-tenant SLA requires it. Evidence: vector-store query event log for the affected period, retrieval-policy-decision log, DLP alert record, embedding-inversion risk assessment.

AI CI/CD pipeline compromise containment: Trigger: ML-Infrastructure CI/CD pipeline integrity failure (unsigned pipeline execution or unauthorized pipeline triggering principal), or ST-Infrastructure pipeline-integrity test failure, or external advisory for a CI/CD component used in AI pipelines. Containment: (1) freeze all AI CI/CD pipelines, set pipeline execution to manual-approval-only mode; (2) audit all pipeline runs in the past 30 days, compare triggering principals against the declared CI/CD service-account allowlist; identify unauthorized pipeline runs; (3) replay eval suite against all model artifacts promoted in the past 30 days from pipelines that may have been compromised; (4) re-verify signatures and SBOM completeness for all model artifacts in the production registry promoted during the affected window; (5) regenerate SBOMs for affected artifacts; (6) notify the supply-chain security contact and assess whether a supply-chain incident notification to sector ISAC is warranted; (7) rotate pipeline runner credentials and signing keys. Evidence: pipeline-run event log, signature-verification event log for affected artifacts, eval-replay results, SBOM regeneration records.

Shadow inference endpoint containment: Trigger: ML-Infrastructure shadow inference endpoint detection (new endpoint in cloud API discovery not in SM-Infrastructure inventory), or egress allow-list alert on outbound traffic to an undeclared LLM provider domain from a new infrastructure component. Containment: (1) apply egress block on the identified shadow endpoint, block outbound traffic from the endpoint's workload identity to the undeclared LLM provider domain via network policy; (2) identify the endpoint's owning team and component from cloud resource tagging or the SM-Infrastructure inventory (or open a new inventory record if absent); (3) route the component through the SM-Infrastructure intake process; (4) conduct a data-flow assessment, determine whether any regulated data transited the LLM provider endpoint (inference request log review, egress log review); (5) if regulated data transited an LLM provider endpoint without a valid DPA or processor agreement on file, evaluate GDPR Art. 33 and EU AI Act Art. 73; (6) evaluate whether the endpoint is operating under a valid EH-Infrastructure hardening baseline. Evidence: shadow-endpoint detection alert record, egress-block configuration record, SM-Infrastructure inventory record creation, data-flow assessment, regulatory evaluation record.

C) Track regulatory SLAs and run post-incident reviews

Regulatory SLA tracker, live, named obligation, with automated escalation on approach:

• GDPR Art. 33, 72-hour supervisory-authority notification window after the controller becomes aware of a personal data breach; clock starts on the first internal alert that constitutes awareness (ML detection, IR finding, external notification). Any AI infrastructure incident involving personal data triggers an immediate GDPR Art. 33 assessment; named owner: Privacy/Legal. The IM-Infrastructure backlog record is flagged with the clock start time; a daily status update is required until the notification is filed or the clock expires.
• EU AI Act Art. 73, serious incident involving a high-risk AI system (Annex III); reporting timeline per the implementing act; track and escalate to Privacy/Legal immediately on any Annex III-hosted-artifact incident. Named owner: Privacy/Legal + executive sponsor.
• HIPAA breach notification, 60-day discovery-to-notification ceiling for covered entities; applies to AI infrastructure processing PHI; flag any AI infrastructure incident involving PHI immediately. Named owner: Privacy/Legal.
• NYDFS Part 500, 72-hour notification to the Superintendent for material cybersecurity events affecting covered entities; applies when AI infrastructure incidents rise to material cybersecurity event threshold. Named owner: CISO / Privacy/Legal.
• PCI-DSS, cardholder data breach notification requirements; applies to AI infrastructure components in the cardholder data environment; named owner per the org's PCI compliance program.
• FedRAMP IR, incident reporting to FedRAMP ISSO and JAB within 1 hour of detection for high-severity incidents (US government cloud deployments); named owner: Cloud Security / FedRAMP ISSO.
• ISO/IEC 27035, incident management process alignment; used as the procedural baseline for all AI infrastructure incident response; named owner: CISO / Infrastructure Security.

Every Critical or blocker incident receives a post-incident review within 14 days of containment: - What happened: root cause, how the incident initiated, what controls failed or were absent. - What caught it: which ML detection, IM source, or external report surfaced it first; was this the expected detection path or a gap? - What did not catch it: which controls should have detected or prevented this but did not. - Update outputs (all four must be populated for Critical incidents): - SA-Infrastructure: pattern-update request if the incident exploited an architectural gap. - SR-Infrastructure: requirements-pack update request if the incident exploited a missing or vague requirement. - EG-Infrastructure: training-content update request if the incident indicates a literacy gap in the infrastructure team. - ML-Infrastructure: detection-update request (new detection, tuned query, or sharpened existing detection query).

Post-incident review outputs are tracked as IM-Infrastructure issues of their own (type: improvement); they age against the same process metric cadence as other issues.

Outcome Metrics (L1)

Process Metrics (leading)

• Backlog triage cadence honored, daily Critical/High triage; weekly Medium; monthly aging.
• Playbook runbook rehearsal, at least one tabletop per quarter exercising an AI-infrastructure-specific incident scenario (rotate through the seven playbook classes).
• Regulatory SLA tracker reviewed weekly; named owner confirms clock start dates and status.
• Aging pockets, number of issues aging past SLA tracked; trending down.

Effectiveness Metrics (business value)

• Repeat-class incident rate, an incident class occurring twice in 12 months that did not produce a SA/SR/EG/ML update after the first occurrence is a process failure; repeat rate on same class trending down.
• Deployer-duty evidence chain, on a regulatory inquiry, the incident records show the logging, containment, and notification chain for the affected infrastructure component; evidence assembled within ≤5 business days.
• Mean-time-to-contain across Critical and High-severity AI infrastructure incidents trending down over quarters.

Success Criteria

• Single AI infrastructure issue backlog established with standardized metadata; triage rubric with AI-infrastructure-specific severity definitions published.
• Seven AI-infrastructure-specific incident playbook entries published (cross-tenant breach, model registry compromise, GPU residual-state leakage, orchestrator compromise, vector-store mass-extraction, AI CI/CD pipeline compromise, shadow inference endpoint), each with named roles, containment steps, evidence-capture instructions, and SLA targets.
• Regulatory SLA tracker live covering GDPR Art. 33, EU AI Act Art. 73, HIPAA, NYDFS Part 500, PCI-DSS, FedRAMP IR, and ISO/IEC 27035; 100% adherence in the last 90 days.
• Post-incident review loop wired to SA-Infrastructure, SR-Infrastructure, EG-Infrastructure, and ML-Infrastructure; every Critical/blocker incident produces a review within 14 days with named update outputs.

Maturity Level 2

Objective: Calibrate incident response depth per SM-Infrastructure L2 tier (Critical / High / Medium / Low); establish dedicated on-call rotation for Critical-tier infrastructure components; and automate cross-domain signal flow so that Infrastructure incidents affecting Software and Data domains generate coordinated response

At this level, incident response differentiates by tier. Critical-tier infrastructure components, the inference cluster serving regulated data, the multi-tenant model registry, the GPU fleet running high-classification workloads, have a dedicated on-call rotation, pre-staged executive escalation paths, and 24/7 coverage. High-tier components have scoped response with defined escalation. Medium and Low follow the standard queue. Post-incident reviews auto-feed SA-Infrastructure, SR-Infrastructure, EG-Infrastructure, and ML-Infrastructure queues via integration rather than manual handoff. When an AI infrastructure incident affects cross-domain scope (an inference endpoint compromise that affects the Software domain, a GPU residual-state leak that exposes Data-domain content, or a vector-store breach that triggers Processes-domain downstream impact), coordinated cross-domain response is activated.

Dependencies

• IM-Infrastructure L1 (required): unified backlog, AI-infrastructure-specific playbook, and regulatory SLA tracker must be operational before per-tier calibration adds meaningful depth.
• SM-Infrastructure L2 (required): tier rubric (Critical / High / Medium / Low) and tier-treatment matrix drive response intensity; without SM L2 tiers, per-tier incident response has no substrate.
• ML-Infrastructure L2 (required): richer detections (anomaly-based, cross-archetype correlated) feed severity classification with higher-fidelity signals; tier-calibrated logging depth makes evidence collection faster at L2.
• Supports / unblocks: ML-Infrastructure L2 detection tuning loop (IM L2 post-incident reviews, now with tier-context, produce more targeted detection updates); SA-Infrastructure L2 pattern evolution (incidents from Critical-tier components with IaC-encoded patterns drive conformance-test updates).

Desired Outcomes

• Response intensity matches tier, Critical-tier infrastructure incidents do not wait in the general SRE queue; they activate a named response team, a dedicated on-call path, and the full containment playbook within the published SLA.
• Post-incident review outputs auto-flow to SA-Infrastructure, SR-Infrastructure, EG-Infrastructure, and ML-Infrastructure backlogs via integration, no update gets lost in a document.
• Cross-domain coordination is explicit: an Infrastructure incident that implicates the Software domain (an inference endpoint compromise that exposes a Software artifact's prompt/completion logs) or the Data domain (a GPU residual-state leak that exposes training corpus content) activates a named cross-domain coordination protocol.
• Tier-movement in the SM-Infrastructure inventory auto-triggers IM policy changes: when a component is re-tiered to Critical, the on-call path, playbook variant, and SLA targets are automatically updated.

Activities

A) Tier-calibrated incident playbook and on-call

Extend L1 playbook entries with tier-specific activation criteria:

• Critical tier: full IM activation, CISO or delegate + Privacy/Legal + infrastructure deployer-duty owner + executive sponsor notification; ≤1 hour acknowledgement; ≤4 hours containment-action initiated; 24/7 on-call coverage with a named AI infrastructure incident responder in each on-call rotation; pre-staged communication templates (internal, customer-facing, regulatory) reviewed quarterly; dedicated infra-on-call rotation briefed with the current Critical-tier component list and their active detection set.
• High tier: scoped response team, Infrastructure Security lead + Privacy/Legal (if regulated data involved) + deployer-duty owner; ≤4 hours acknowledgement; ≤24 hours containment-action initiated; business-hours on-call with after-hours escalation path.
• Medium tier: standard response; ≤1 business day acknowledgement; queue-based triage.
• Low tier: tracked in queue; aggregated weekly handling.

B) Post-incident review auto-flow integration

Wire IM-Infrastructure's post-incident review outputs to downstream practice backlogs via a defined integration:

• SA-Infrastructure pattern-update request → SA-Infrastructure architecture-backlog ticket (auto-created with IM incident reference linked).
• SR-Infrastructure requirements-pack update request → SR-Infrastructure pack-backlog ticket (auto-created with requirements-pack version and failing requirement row linked).
• EG-Infrastructure training-content update request → EG-Infrastructure training-backlog ticket (auto-created with affected population segment and incident summary linked).
• ML-Infrastructure detection-update request → ML-Infrastructure detection-registry update ticket (auto-created with detection name, current query, and proposed change linked).

SLA for downstream updates: Critical-tier post-incident review outputs must be accepted or rejected by the downstream practice owner within 14 days. Accepted updates are treated as High-severity issues in the receiving practice's backlog.

C) Cross-domain coordination protocol

Publish a cross-domain coordination protocol that activates when an Infrastructure-domain AI incident implicates another domain:

• Infrastructure → Software: an inference endpoint compromise exposes a Software-domain artifact's prompt/completion logs or model version; activates Software-domain EH and IM alongside Infrastructure-domain containment. Named Software-domain IM contact on file.
• Infrastructure → Data: a GPU residual-state leak or vector-store mass-extraction exposes Data-domain training corpus or inference-input content; activates Data-domain IM alongside Infrastructure-domain containment plays for residual-state leakage or vector-store mass-extraction. Named Data-domain IM contact on file.
• Infrastructure → Processes: an orchestrator control-plane compromise causes unauthorized writes to a business-process workflow; activates Processes-domain business-continuity coordinator alongside Infrastructure-domain orchestrator-compromise play. Named Processes-domain contact on file.

Cross-domain incident activations: shared status board, one unified IC (from the primary impacted domain), coordinated remediation tracking, joint post-incident review spanning all affected domains. Infrastructure-incident-driven SA/SR/EG/ML updates auto-flow across all affected domains via the integration from Activity B.

Outcome Metrics (L2)

Process Metrics (leading)

• Critical-tier playbook review cadence, quarterly, tested in a tabletop covering the tier's specific component list.
• On-call rotation health, no uncovered periods; hand-off briefing includes updated Critical-tier component list; clearing-failure and shadow-endpoint escalation paths rehearsed quarterly.
• Post-incident review quality reviewed quarterly by the program sponsor, are update outputs substantive (concrete change to a pattern, pack, curriculum, or detection)?
• Cross-domain coordination contacts verified quarterly, named contacts are current, communication channels tested.

Effectiveness Metrics (business value)

• Dwell time on Critical-tier infrastructure incidents (first ML detection to containment action complete) trending down as L2 matures.
• Downstream practice update acceptance rate, % of Critical-tier post-incident updates accepted and resolved by the downstream practice; measures whether the feedback loop improves the program.
• Cross-domain coordination saves time vs. uncoordinated parallel response, measured as MTTU and MTTC on multi-domain incidents.

Success Criteria

• Critical-tier MTTA ≤1 hour; MTTC ≤4 hours; 24/7 on-call coverage with a documented rotation and a current Critical-tier component briefing.
• Post-incident review auto-flow integration live; 100% of Critical-tier review outputs auto-routed to SA/SR/EG/ML backlogs; ≥90% of downstream practice owners responding within 14 days.
• Cross-domain coordination protocol published and used for 100% of multi-domain AI infrastructure incidents.
• Tier-movement in SM-Infrastructure inventory auto-triggers IM configuration update (on-call path, playbook variant, SLA targets) within 14 days (Critical re-tier) or 30 days (other tiers).

Maturity Level 3

Objective: Contribute incident patterns and playbook templates to CNCF, OpenSSF AI, MITRE ATLAS, AVID, and sector ISACs; automate runbook decisioning for low-severity, high-confidence detections; and benchmark MTTR against industry peers

At this level, IM-Infrastructure is a contributor to the AI-infrastructure incident-response ecosystem. Anonymized incident classification schemes, AI-infrastructure-specific severity anchors, and playbook templates for the seven primary AI infrastructure incident classes are contributed to CNCF TAG Security, OpenSSF AI Infrastructure, AVID, MITRE ATLAS, and sector ISACs. Pre-authorized automated containment actions execute for low-severity, high-confidence detections without human triage delay. MTTR benchmarks are established from ISAC and peer data and the program's MTTR is measured against them quarterly.

Dependencies

• IM-Infrastructure L2 (required): tiered playbook, post-incident review auto-flow, cross-domain coordination must be operational and producing clean incident-pattern data before external contributions are substantive.
• PC-Infrastructure L3 (required): continuous compliance attestation substrate supports automated evidence capture for pre-authorized containment actions; legal authority for automated actions flows from the policy and compliance program.
• ML-Infrastructure L3 (required): detection-as-code and high-confidence anomaly detection signals provide the automation trigger quality needed for pre-authorized runbook execution.

Desired Outcomes

• Industry-standard incident classification and response playbooks for AI infrastructure incidents are contributed and maintained, CNCF, OpenSSF AI, AVID, and MITRE ATLAS cite the org's artifacts.
• Pre-authorized automated containment actions execute for a defined set of low-severity, high-confidence incident types, reducing MTTR for these classes to seconds from hours.
• MTTR benchmarks established from ISAC and peer data; program performance reported to the sponsor quarterly and drives investment decisions.
• Contributions to MITRE ATLAS AI infrastructure tactic documentation reflect the org's first-party incident experience.

Activities

A) Industry-coordinated incident sharing and contribution

• Participate in sector ISAC AI incident-sharing programs (FS-ISAC AI working group, H-ISAC, IT-ISAC, sector-specific):
• Consume ISAC AI incident feeds; integrate relevant advisories into the IM-Infrastructure external-advisory source.
• Contribute anonymized incident classification (incident type, archetype affected, ATLAS tactic tag, containment play used, MTTR achieved) on a per-incident-class basis; target ≥4 ISAC contributions per year.
• Contribute to AI infrastructure incident taxonomy standards:
• CNCF TAG Security, AI infrastructure incident severity-anchor definitions, playbook template schemas for the seven AI infrastructure incident classes, Kubernetes-specific containment runbook patterns.
• OpenSSF AI Infrastructure Working Group, supply-chain incident response playbooks for model registry compromise and CI/CD pipeline integrity failure; contribute to OpenSSF AI incident taxonomy.
• AVID (AI Vulnerability Database), submit AI infrastructure vulnerability entries for novel incident classes discovered in production (GPU residual-state leakage vectors, orchestrator injection surfaces, vector-store extraction patterns); target ≥2 AVID entries per year.
• MITRE ATLAS, contribute incident-derived technique observations or mitigation entries for AI infrastructure TTPs (cross-tenant access, model supply-chain compromise, orchestrator workflow injection); target ≥1 ATLAS contribution per year.

B) Pre-authorized automated runbook decisioning

Define and publish a pre-authorization policy for automated containment actions, vetted by Privacy/Legal and the executive sponsor:

• Pre-authorized actions (published list):
• Egress-block for shadow inference endpoint (first-time detection of new inference endpoint not in SM-Infrastructure inventory for non-Critical-tier components).
• GPU node drain on residual-state clearing failure for Medium-tier or lower nodes (drain + clearing-audit trigger, no workload restart until human confirms clearing succeeded).
• Vector-store retrieval-policy block for a principal exceeding the mass-extraction threshold on a non-Critical-tier store (rate-limit the principal's query access).
• Pipeline execution freeze for Low/Medium-tier AI CI/CD pipelines on pipeline integrity failure detection (set to manual-approval-only mode pending investigation).
• Pre-authorized actions for Critical-tier components require human confirmation within 15 minutes; the action fires after that window if no human confirmation arrives (timer-based fallback), with executive notification at fire time.
• All pre-authorized actions produce: a full audit log entry in the IM-Infrastructure backlog, a human-review ticket auto-created at execution time, and a notification to the component's deployer-duty owner.
• Pre-authorization policy reviewed quarterly by Privacy/Legal and the executive sponsor; any automated action that produces an unexpected outcome triggers a review of the pre-authorization threshold.

C) MTTR benchmarking

• Establish MTTR benchmarks from ISAC AI incident data exchanges, BSIMM-style observational data on AI infrastructure incident response at comparable organizations, MITRE ATLAS practitioner community data, and peer roundtables.
• Publish a quarterly MTTR benchmark brief to the program sponsor covering MTTR per incident class (cross-tenant breach, model registry compromise, GPU residual-state leakage, orchestrator compromise, vector-store mass-extraction, CI/CD pipeline compromise, shadow inference endpoint), MTTR per tier, and delta trend vs. benchmark.
• Where MTTR is above benchmark, root-cause is mapped to a specific practice gap with a budget-linked improvement proposal.

Outcome Metrics (L3)

Process Metrics (leading)

• ISAC participation cadence, sector ISAC feeds consumed and contributions submitted at least quarterly; ISAC AI exercises attended annually.
• Contribution pipeline health, ≥2 taxonomy/playbook/ATLAS items in-flight (draft, in-review, submitted) at any time.
• Pre-authorization policy review cadence, quarterly; any automated action producing an unexpected outcome triggers an out-of-cycle review.
• Benchmark data source refresh, MTTR benchmark inputs updated at least semi-annually.

Effectiveness Metrics (business value)

• MTTR for pre-authorized containment classes (shadow endpoint egress-block, GPU node drain, vector-store rate-limit) drops to seconds from hours.
• Mean-time-to-contain on Critical-tier infrastructure incidents continuing to compress as ISAC-shared intelligence accelerates root-cause identification.
• External recognition, citations or adoption of contributed AI infrastructure incident taxonomy artifacts by CNCF, OpenSSF AI, AVID, ATLAS, or sector ISACs.

Success Criteria

• ≥4 ISAC AI incident contributions per year; ≥2 AVID entries per year; ≥1 ATLAS AI infrastructure contribution per year; all contributions anonymized, legally vetted, and maintained.
• ≥3 pre-authorized automated containment actions live, vetted by Privacy/Legal and the executive sponsor, producing 100% audit records + human-review tickets on execution.
• Quarterly MTTR benchmark brief published to sponsor; Critical-tier MTTR at or below benchmark for ≥4 of 7 incident classes; deltas above benchmark linked to investment proposals.
• Pre-authorization policy reviewed quarterly; no unauthorized automated action executed; all unexpected automation outcomes reviewed within 5 business days.

Key Success Indicators

Level 1: - Single AI infrastructure issue backlog operational with standardized metadata (source, affected component linked to SM-Infrastructure inventory, severity rubric anchored to AI-infrastructure-specific axes, owner, SLA, regulatory flag, evidence link) capturing ≥95% of AI infrastructure issues from all source practices. - Seven AI-infrastructure-specific incident playbook entries published (cross-tenant breach, model registry compromise, GPU residual-state leakage, orchestrator compromise, vector-store mass-extraction, AI CI/CD pipeline compromise, shadow inference endpoint) with named roles, containment plays, evidence-capture instructions, and SLA targets, each exercised in at least one tabletop in the last 12 months. - Regulatory SLA tracker live covering GDPR Art. 33 (72h), EU AI Act Art. 73, HIPAA (60d), NYDFS Part 500 (72h), PCI-DSS, FedRAMP IR (1h high-severity), and ISO/IEC 27035; 100% adherence in the last 90 days. - Post-incident review loop wired to SA-Infrastructure, SR-Infrastructure, EG-Infrastructure, and ML-Infrastructure, every Critical/blocker incident produces a review within 14 days with named update outputs for each downstream practice.

Level 2: - Critical-tier MTTA ≤1 hour; MTTC ≤4 hours; 24/7 on-call coverage with a documented rotation that includes a current Critical-tier component briefing and clearing-failure/shadow-endpoint escalation paths. - Post-incident review auto-flow integration live; 100% of Critical-tier review outputs auto-routed to SA/SR/EG/ML backlogs; ≥90% of downstream practice owners responding within 14 days. - Cross-domain coordination protocol published and used for 100% of multi-domain AI infrastructure incidents; named cross-domain contacts for Software, Data, and Processes domains verified quarterly. - Tier-movement in SM-Infrastructure inventory auto-triggers IM configuration updates within 14 days (Critical re-tier) or 30 days (other tiers).

Level 3: - ≥4 ISAC contributions per year; ≥2 AVID entries per year; ≥1 ATLAS AI infrastructure contribution per year, all maintained and tracked for external adoption. - ≥3 pre-authorized automated containment actions live, vetted, producing 100% audit records, with quarterly policy review and zero unauthorized executions. - Quarterly MTTR benchmark brief published; Critical-tier MTTR at or below benchmark for ≥4 of 7 incident classes; deltas linked to investment proposals.

Common Pitfalls

Level 1: - ❌ "Single backlog" created but ML-Infrastructure detection alerts continue routing to the SRE on-call Slack channel and GPU clearing failures are treated as platform reliability events rather than security issues, the backlog captures DR and IR findings but misses the runtime signals that are often the first indicator of an active infrastructure attack. - ❌ Triage rubric severity anchors use generic cloud-security axes, a GPU residual-state clearing failure between tenants (which can be Critical depending on workload classification) is triaged Low because "no service disruption" and the cross-tenant exposure is not recognized as the severity driver. - ❌ Playbook entries published for model registry compromise and shadow inference endpoint but roles are not pre-assigned, on the first live model registry compromise, the team spends the first hour figuring out whether the infrastructure security team or the ML platform team owns the promotion freeze, not freezing it. - ❌ GDPR Art. 33 72-hour clock tracked informally, when a cross-tenant access detection fires on an inference endpoint processing EU data subjects' inputs, the clock starts but no named owner confirms the awareness event start time; the SLA slips over a weekend before anyone documents it. - ❌ Post-incident reviews completed for Critical incidents but update outputs are filed in a Confluence page that no downstream practice owner reads, SA-Infrastructure, SR-Infrastructure, and ML-Infrastructure do not update; the same cross-tenant isolation gap is exploited in a second incident six months later. - ❌ External CVE advisories for AI infrastructure components (vLLM, Triton, Weaviate, Apache Airflow in the ML pipeline) are not routed to IM-Infrastructure, they arrive in the cloud-security team's advisory feed and sit unprocessed because the AI infrastructure component is not in the cloud-security team's scope; the IM-Infrastructure backlog never sees them.

Level 2: - ❌ Critical-tier activation criteria are vague, a GPU residual-state clearing failure on a Critical-tier node initially triaged as a platform reliability issue sits in the SRE queue for 6 hours before someone recognizes the security implication; the GDPR Art. 33 clock has been running since the ML detection fired. - ❌ Post-incident review auto-flow integration wired but downstream practice backlogs never treat the auto-created tickets as actionable, the SR-Infrastructure team closes the ticket as "acknowledged" without updating the requirements pack; the cross-tenant isolation requirement gap recurs in the next new inference cluster deployment. - ❌ Cross-domain coordination protocol exists on paper but no IC is pre-designated for Infrastructure→Data incidents, the first vector-store mass-extraction that also constitutes a Data-domain breach produces ownership confusion; the Data-domain IM and Infrastructure-domain IM both wait for the other to confirm the GDPR Art. 33 clock start. - ❌ 24/7 on-call coverage implemented but the on-call briefing is stale, the rotation includes a Critical-tier component list that was accurate 90 days ago; a new inference cluster promoted to Critical-tier during a compliance re-assessment is not in the briefing; on-call responders do not know the escalation path or the clearing-failure playbook for it.

Level 3: - ❌ ISAC participation limited to consuming feeds, contributions are absent; influence over AI infrastructure incident taxonomy standards diminishes; the ISAC feed quality degrades without reciprocal intelligence from the org's first-party GPU residual-state and orchestrator injection experience. - ❌ Pre-authorized automated containment fires on a Critical-tier inference endpoint because the confidence threshold was set too loosely, a false positive from the vector-store mass-extraction detection triggers an egress block on a Critical-tier production inference cluster serving regulated data; the pre-authorization policy had no Critical-tier exception check. - ❌ MTTR benchmark brief cites benchmarks from organizations with fundamentally different AI infrastructure portfolio scale, "we are at benchmark" is true but the benchmark set was chosen to flatter rather than stretch. - ❌ AVID entries submitted once and never updated, GPU residual-state leakage vectors evolve as new accelerator architectures are deployed; the org's AVID entry reflects a CUDA-based clearing pattern that does not apply to the next-generation accelerators now in the fleet; the community builds on stale guidance. - ❌ Automated containment produces audit records that are technically complete but lack the narrative context needed for post-incident root-cause review, humans reviewing automated GPU-drain logs cannot reconstruct what the clearing-failure event saw or why the anomaly threshold triggered, making it impossible to distinguish a true leakage event from a DaemonSet timing issue.

Practice Maturity Questions

Level 1: 1. Is there a single AI infrastructure issue backlog with standardized metadata (source, affected component linked to SM-Infrastructure inventory, severity rubric anchored to AI-infrastructure-specific axes, cross-tenant breach / GPU residual-state leakage / registry compromise for Critical; confirmed control failure with potential impact for High, etc., owner, SLA, regulatory flag, evidence link) capturing ≥95% of issues from all source practices (TA, SR, DR, IR, ST, ML, external CVEs, CNCF advisories, ATLAS updates)? 2. Is the AI infrastructure incident playbook published with ≥7 named AI-infrastructure-specific incident classes (cross-tenant breach, model registry compromise, GPU residual-state leakage, orchestrator compromise, vector-store mass-extraction, AI CI/CD pipeline compromise, shadow inference endpoint), each with pre-assigned roles, containment plays, evidence-capture steps, and SLA targets, and has each class been exercised in at least one tabletop in the last 12 months? 3. Is the regulatory SLA tracker live covering GDPR Art. 33 (72h), EU AI Act Art. 73, HIPAA (60d), NYDFS Part 500 (72h), PCI-DSS, FedRAMP IR (1h high-severity), and ISO/IEC 27035, with 100% adherence in the last 90 days, and does every Critical/blocker incident produce a post-incident review within 14 days with named update outputs flowing to SA-Infrastructure, SR-Infrastructure, EG-Infrastructure, and ML-Infrastructure?

Level 2: 1. Is a tier-calibrated incident playbook operational with Critical-tier MTTA ≤1 hour and MTTC ≤4 hours, 24/7 on-call coverage with a documented rotation including a current Critical-tier component briefing and rehearsed escalation paths for GPU clearing failure and shadow endpoint, and tier-movement in SM-Infrastructure inventory automatically triggering IM configuration updates within 14 days (Critical re-tier)? 2. Is a post-incident review auto-flow integration live routing Critical-tier review outputs to SA/SR/EG/ML practice backlogs, with ≥90% of downstream practice owners responding within 14 days and the sponsor reviewing output quality quarterly to distinguish substantive changes from nominal acknowledgements? 3. Is a cross-domain coordination protocol published and used for 100% of multi-domain AI infrastructure incidents, with named cross-domain contacts for Software, Data, and Processes domains verified quarterly, a single IC from the primary impacted domain, and joint post-incident reviews spanning all affected domains?

Level 3: 1. Does the program contribute ≥4 anonymized AI infrastructure incident-classification entries per year to sector ISACs, ≥2 entries per year to AVID, and ≥1 contribution per year to MITRE ATLAS AI infrastructure tactic documentation, with all contributions maintained current, legally vetted, and tracked for external adoption? 2. Are ≥3 pre-authorized automated containment actions live (shadow-endpoint egress-block, GPU node drain on clearing failure, vector-store retrieval rate-limit, or AI CI/CD pipeline freeze classes), vetted by Privacy/Legal and the executive sponsor, producing 100% audit records plus human-review tickets on execution, with the pre-authorization policy reviewed quarterly and any unexpected outcome triggering an out-of-cycle review? 3. Is a quarterly MTTR benchmark brief published to the sponsor comparing the program's MTTR per incident class and per tier against ISAC-sourced and peer-sourced benchmarks, with Critical-tier MTTR at or below benchmark for ≥4 of 7 incident classes and deltas above benchmark linked to specific practice gaps and investment proposals?

Document Version: HAIAMM v3.0, 2026-05-14, Verifhai Practice: Issue Management (IM) Domain: Infrastructure Last Updated: 2026-05-14 Author: Verifhai