# Monitoring & Logging (ML)
## Software Domain - HAIAMM v3.0

---

### Practice Overview

**Objective:** Establish the **logging baseline per AI/HAI software archetype**, operate a **small high-signal detection set** targeted at the top threats from TA-Software, and produce the **evidence trail** that proves EU AI Act deployer duties, GDPR processor obligations, and ISO/IEC 42001 AIMS requirements, on demand, inside a published SLA.

**Description:** ML-Software captures the signals produced by every AI/HAI software artifact the organization **builds**, LLM-integrated applications, autonomous agents, RAG pipelines, fine-tuning and training workloads, evaluation harnesses, model-serving services, and classical ML models in product surfaces. For each archetype it specifies the exact events to capture (prompt/completion, tool-call, admin-audit, identity), the retention window required to satisfy the longest applicable regulation, and the export path that supports auditor review. On top of the logging baseline it operates a bounded, purposeful detection set, each detection tied to a threat from TA-Software, with a named owner, a defined query, and an active tuning record. The full corpus produced by ML-Software is the primary evidence artifact for PC-Software's compliance map: EU AI Act Art. 12 deployer-duty logs, GDPR Art. 30 records of processing, and ISO/IEC 42001 AIMS operational evidence.

**Context:** Logging AI/HAI software is not the same as logging classic web services. A prompt/completion event requires request-id correlation, model version, classification label, and latency alongside text content or a hash. A tool-call event from an agent must carry the tool name, arguments, return value, principal, and success/fail status, not just an HTTP status code. Training-job events must capture data-source lineage and eval-gate results to support model-promotion audit. None of this exists by default in standard APM or SIEM tooling unless someone has explicitly instrumented the archetype's event schema. ML-Software makes that schema explicit, per archetype, from day one, so the organization is not reconstructing an evidence trail from incomplete telemetry the first time a regulator or incident demands it.

---

## Maturity Level 1
### Objective: Establish the per-archetype logging baseline, operate a small high-signal detection set targeting the top TA-Software threats, and produce an on-demand evidence trail that satisfies EU AI Act Art. 12, GDPR Art. 30, and ISO/IEC 42001 AIMS requirements within a published SLA

At this level, the program makes every production AI/HAI software artifact observable with a defined minimum event schema, closes the most dangerous detection gaps (AGH, prompt-injection success, tool-scope violations, shadow-AI emergence), and demonstrates that the resulting log corpus can produce deployer-duty evidence for auditors and regulators on demand.

#### Dependencies

- **SM-Software L1 (required):** the inventory and archetype taxonomy define what must be instrumented; ML-Software L1 cannot baseline what SM has not yet catalogued.
- **SA-Software L1 (required):** reference patterns specify where logs originate (API gateway, proxy layer, agent executor, training pipeline, model-serving endpoint); the patterns specify what to log.
- **EH-Software L1 (required):** hardened identity controls (SSO, service-principal model, secrets vault) produce the identity events that ML captures; without EH instrumentation there are no usable identity-event log sources.
- **TA-Software L1 (required):** archetype threat library drives detection priority; the high-signal detection set is selected from TA's top threat list, not invented independently.
- **Supports / unblocks:** IM-Software L1 (detections become the primary input to the issue backlog); PC-Software L1 (ML log retention and export path constitutes the compliance evidence trail the PC compliance map requires).

#### Desired Outcomes

- Every production AI/HAI software artifact emits the per-archetype minimum event schema; no artifact produces logs that are architecturally insufficient for deployer-duty evidence.
- Retention meets or exceeds the longest applicable compliance window across the active regulatory set; export path demonstrated for each archetype at least annually.
- A small, bounded detection set is live, each detection tied to a named TA-Software archetype threat, each with an owner, a query, a SLA, and a tuning record; false-positive rate tracked per detection.
- EU AI Act Art. 12 high-risk-system log obligations, GDPR Art. 30 records-of-processing entries, and ISO/IEC 42001 AIMS evidence assets can be satisfied from ML-Software log stores within the published SLA (on-demand pull ≤24 hours).
- Shadow-AI emergence in the engineering environment is detectable from ML signals alone (new outbound flows to LLM provider domains from unregistered services surface immediately in the detection set).

#### Activities

**A) Establish the per-archetype logging baseline**

Define and instrument the minimum event schema for each archetype in the SM-Software inventory. Each event record includes: request-id / correlation-id, principal (user or service account), timestamp, archetype tag, and the archetype-specific fields below. PII scrubbing applied per SR-Software data-boundary requirements before logging.

**LLM-integrated application:**
- Prompt event: request-id, user/principal, prompt text or hash (hash where regulated data may be present), classification label, retrieval sources (if RAG-backed), model + version, timestamp.
- Completion event: request-id, completion text or hash, model + version, input tokens, output tokens, latency ms, error code.
- Guardrail-decision event: filter triggered (name), reason, action taken (pass / block / redact), request-id.
- Rate-limit / abuse-detection event: principal, rate-rule triggered, action.

**AI agent:**
- All LLM-integrated-app events above.
- Tool-call event: tool name, full arguments (or argument hash for sensitive parameters), return value or hash, principal, timestamp, success/fail, latency ms.
- Agent-goal vs. action delta event: declared goal, observed tool invocation, delta flag (triggered when action is outside declared goal scope).
- HITL-gate event: gate triggered, approver identity, decision (approved / declined), timestamp.
- Kill-switch event: trigger source, timestamp, agent session scope affected, invoking principal.

**RAG pipeline:**
- Retrieval event: sources retrieved (document IDs + classification labels), source provenance, tenant-id for multi-tenant deployments, query-id correlated to the prompt event.
- Injection-defense decision event: retrieved-content flagged (yes/no), reason, action (include / exclude / redact).

**Fine-tuning / model-training workload:**
- Training-job event: job-id, data-source list with classification labels and consent-basis flags, model-output identifier, final eval-results summary, training-job duration.
- Model-promotion event: model-id, eval gates passed (list), approver, timestamp, promoted-to registry location.

**Evaluation harness:**
- Eval-run event: corpus version, model version, result (pass/fail), regression-delta vs. baseline, eval-job-id.

**Model-serving service:**
- Version-deployment event: model version, deployer principal, canary percentage, deployment-id.
- Canary-decision event: canary result, promotion decision, approver.
- Rollback event: trigger reason, from-version, to-version, invoking principal, timestamp.

**Classical ML model:**
- Inference event: model-id, feature-set version, prediction, confidence score, principal.
- Drift-detection event: drift type (data/concept), magnitude vs. baseline, threshold crossed (yes/no), alert routed.

**Admin-audit events (all archetypes):**
- Config changes to the artifact or its infrastructure components.
- Secret rotation events for LLM provider API keys.
- IAM changes to roles and service principals associated with the artifact.
- Model-registry changes (promotion, deprecation, deletion).
- Tool-list changes for agent artifacts (tool added, tool removed, scope modified).
- Kill-switch arm / disarm events.
- On-call hand-off records.

**Identity events (cross-archetype):**
- SSO sign-ins to model registry consoles, LLM provider admin consoles, CI/CD systems, and code repositories.
- Service-principal token use against LLM provider APIs (principal, provider, volume, timestamp).

**Retention and exportability:**
- Retention window: meets or exceeds the longest applicable requirement across active regulations (EU AI Act Art. 12 high-risk logs ≥6 months; GDPR Art. 30 retention per data-class and processing purpose; HIPAA PHI where applicable ≥6 years; sector-specific where applicable). Where multiple retention windows apply to a single artifact, the longest governs.
- Export path: JSON or structured CSV export from the log store tested at least annually; on-demand pull SLA ≤24 hours for evidence requests from auditors, regulators, or legal hold.
- Log integrity: write-once or append-only storage for admin-audit and deployer-duty evidence tiers; access-control separation between application teams and log store administrators.

**B) Operate a small high-signal detection set**

L1 target: ≤12 detections, each tied to a TA-Software archetype threat and to at least one HAI TTP tag (EA / AGH / TM / RA) or ATLAS tactic (TA0001 Reconnaissance used for early-stage detection; TA0014 Impact addressed in IM). Each detection has: owner, detection query, SLA (time-to-IM-ticket), and last-tuned date. False-positive rate tracked per detection; monthly tuning review.

Core detection set:

- **AGH detection** (ATLAS TA0009 Collection / AGH TTP), multi-turn agent goal drift: agent goal declared at session start vs. tool invocations across turns; delta above threshold fires detection.
- **Prompt-injection success detection** (ATLAS TA0013 Exfiltration / AGH TTP), output exfiltration patterns: completion text matching known exfiltration signatures (data-format markers, credential patterns, canary strings) on outbound completions.
- **Tool-scope-violation detection** (ATLAS TA0007 Privilege Escalation / EA TTP + TM TTP), agent attempted a tool invocation with a tool or argument outside the published allowlist; fires on any tool-call event where tool name is not in the artifact's declared allowlist or argument exceeds scoped range.
- **Training-data-leakage canary detection** (ATLAS TA0013 Exfiltration / TM TTP), canary string (injected into training corpus at a known position) emitted verbatim in a completion; signals training-data memorization.
- **Shadow-AI emergence detection** (ATLAS TA0001 Reconnaissance / EA TTP), new outbound network flow to an unsanctioned LLM provider domain originating from a service that does not appear in the SM-Software inventory; fires on first-seen detection per service-domain pair.
- **Kill-switch-not-triggered detection**, incident-state flag active for an agent artifact and kill-switch event not present in the log within the declared response SLA; escalates immediately to IM-Software.
- **HITL-gate-bypass attempt detection** (ATLAS TA0008 Defense Evasion / AGH TTP), HITL-gate event shows repeated decline followed by an attempt to invoke the same destructive tool in the same session without a new gate event.

Each detection routes to the IM-Software backlog on fire; median detection-to-ticket time target ≤1 hour for Critical-tier artifacts.

**C) Produce and drill the deployer-duty evidence trail**

ML-Software is the primary evidence source for PC-Software's compliance map. At L1, wire the log store to the compliance requirements:

- **EU AI Act Art. 12** (high-risk-system logging for deployer duties): for every artifact assessed as Annex III high-risk or carrying a customer-facing decision-affecting output, confirm that prompt/completion, tool-call, and admin-audit events are captured and retained at the required window; produce a deployer-duty evidence view (log record + retention attestation + export test result) for each such artifact.
- **GDPR Art. 30** (records of processing): for every artifact processing personal data, the prompt/completion log entries with principal identity, data-class tag, and purpose label constitute the records-of-processing operational entries; link the log-store retention policy to the Art. 30 record for each artifact.
- **ISO/IEC 42001 AIMS** (operational evidence for the AI Management System): training-job events, model-promotion events, eval-run events, and admin-audit events constitute the AIMS operational records; identify gaps and open IM-Software findings for any archetype not yet emitting these events.

Quarterly deployer-duty drill: pull the deployer-duty evidence package for one randomly selected production AI/HAI software artifact per archetype within the published SLA (≤24 hours from request to assembled package). Record drill results; gaps route to IM-Software.

#### Outcome Metrics (L1)

| Metric | Baseline | L1 Target | Source |
|---|---|---|---|
| % production AI/HAI software artifacts meeting the per-archetype logging baseline | measure | ≥90% within 12 months | Logging configuration audit × SM inventory |
| High-signal detection set published and active | 0 / ≤12 | target set defined + ≤12 active detections | Detection registry |
| Median detection-to-IM-ticket time for Critical-tier artifacts | measure | ≤1 hour | Alert → ticket telemetry |
| Deployer-duty evidence pull time (quarterly drill) | measure | ≤24 hours | Drill records |
| False-positive rate per detection (trend) | measure | tracked per detection; monthly tuning review | Detection tuning log |
| % production AI/HAI artifacts with retention meeting longest applicable regulation | measure | 100% | Retention policy audit × inventory |

#### Process Metrics (leading)

- Archetype-baseline gap list maintained, every production artifact scored against its archetype baseline; gaps on IM-Software backlog with named owner.
- Detection tuning cadence, monthly review per detection; stale or superseded detections retired; new detection candidates from IM post-incident reviews queued.
- Retention / export test cadence, at least annually per archetype, with documented evidence.
- Deployer-duty drill cadence honored, quarterly, covering all archetypes in rotation.

#### Effectiveness Metrics (business value)

- Incidents detected internally before external notification, trend over quarters.
- Regulator / auditor / legal evidence requests turned around inside the published SLA (≤24 hours).
- Detection-to-prevention conversion, detections that led to a preemptive control change (SA, SR, EH update) rather than only an incident finding.

#### Success Criteria

- Per-archetype logging baseline published and instrumented for ≥90% of production AI/HAI software artifacts.
- ≤12-detection high-signal set live, each with owner, detection query, SLA, and monthly tuning record; false-positive rate tracked per detection.
- Retention meets the longest applicable regulatory window for every production artifact; export path tested at least annually.
- EU AI Act Art. 12, GDPR Art. 30, and ISO/IEC 42001 AIMS evidence-trail wiring documented; quarterly deployer-duty drill executed inside the ≤24-hour SLA.

---

## Maturity Level 2
### Objective: Calibrate logging depth and detection set to the SM-Software L2 risk-tier rubric; integrate with SIEM for cross-artifact correlation; and feed incident-driven and ST-driven detection updates into a continuous tuning loop

At this level, monitoring intensity matches risk tier. Critical-tier artifacts receive full prompt/completion and tool-call log corpora retained for the longest regulatory window; Low-tier artifacts receive the baseline only. The SIEM ingests ML-Software log feeds and executes cross-artifact correlation rules. The detection set evolves continuously from IM-Software post-incident reviews and ST-Software findings rather than remaining static. Anomaly detection on prompt/completion/tool-call corpora supplements the rule-based set for Critical and High-tier artifacts.

#### Dependencies

- **ML-Software L1 (required):** per-archetype logging baseline, detection set, and deployer-duty evidence trail must be operational before tier calibration is meaningful.
- **SM-Software L2 (required):** risk-tier rubric and tier-treatment matrix drive per-tier logging depth and detection priority; without SM L2 tiers the calibration has no substrate.
- **EH-Software L2 (required):** hardened control surfaces at L2 produce richer identity and SSO signals; ML-Software L2 consumes them for anomaly baselining.
- **IM-Software L1+ (required):** post-incident reviews from IM feed detection-tuning and new-detection requests into ML's review cycle.
- **ST-Software L1+ (required):** security test findings (CI corpus failures, red-team outputs) identify gaps in the detection set and drive new detection candidates.
- **Supports / unblocks:** PC-Software L2 (tier-calibrated compliance evidence bundles auto-assembly requires ML L2 log completeness signals); IM-Software L2 (richer detections feed the tiered incident playbook with higher-fidelity severity signals).

#### Desired Outcomes

- Logging intensity visibly differentiates by tier: Critical-tier artifacts retain full prompt/completion and tool-call corpora for the longest regulatory window; Low-tier artifacts produce only the baseline schema.
- SIEM integration enables cross-artifact correlation, a single attacker session interacting with multiple AI/HAI artifacts in sequence surfaces as a correlated detection rather than isolated per-artifact alerts.
- The detection set evolves quarterly from a defined, governed feedback loop (IM post-incident reviews + ST findings + external advisory updates from ATLAS, OWASP LLM, AVID) rather than by ad hoc addition.
- Anomaly detection establishes behavioral baselines for Critical and High-tier artifacts; rule-based detections that fire frequently without incident are refined or retired before the L2 cycle closes.

#### Activities

**A) Tier-calibrated logging depth**

Apply the SM-Software L2 tier-treatment matrix to logging configuration:

- **Critical tier:** full prompt text and completion text (not hashes) retained for the longest regulatory window; full tool-call argument and return corpora retained; admin-audit and identity events at maximum fidelity; all detections tuned to the artifact; per-tenant isolation enforced at the log store (Critical-tier artifact logs partitioned from other tier logs).
- **High tier:** full prompt/completion text retained; tool-call events at full fidelity; standard admin-audit and identity events; core detections active.
- **Medium tier:** prompt/completion hashes retained for regulatory window; standard admin-audit; shadow-AI emergence + baseline detections active.
- **Low tier:** baseline logging schema only; shadow-AI emergence detection only.

For every Critical-tier artifact, the ML log store is the primary source for PC-Software's compliance evidence bundle (per PC-Software L2 staleness thresholds: ML logging-baseline validation ≤30 days for Critical).

**B) SIEM integration and cross-artifact correlation**

- Ingest all tier-appropriate ML-Software log feeds into the SIEM.
- Author and maintain at least three cross-artifact correlation rules at L2:
  - Multi-artifact AGH correlation: the same principal appears in HITL-gate-bypass attempts on two or more agent artifacts in the same session window, fires a unified incident.
  - Training-to-inference leakage chain: a canary-string detection in a completion correlates to a training-data-leakage canary detection from the same model-id's training-job event, escalates to Critical regardless of artifact tier.
  - Shadow-AI emergence + identity pivot: a shadow-AI emergence detection on a service correlates to an unusual SSO sign-in to an LLM provider admin console from the same service-principal in the same time window.
- Cross-artifact correlation alerts route to IM-Software at the tier of the highest-tier artifact involved.

**C) Detection tuning loop: IM post-incident and ST feedback**

Operate a quarterly detection review cycle:
- IM-Software post-incident reviews that touch a logging or detection gap generate a detection-update request (new detection, tuned query, or retired false-positive rule).
- ST-Software CI corpus failures (prompt-injection regression, data-egress canary, kill-switch test) that are not caught by the current detection set generate a detection-gap finding routed to ML-Software.
- External advisory updates (MITRE ATLAS new techniques, OWASP LLM Top 10 updates, AVID advisories) are assessed quarterly; each applicable update either adds a new detection candidate or updates an existing detection's query.
- Monthly anomaly-baseline refresh for Critical and High-tier artifacts: normal behavior baseline (prompt volume, tool-call patterns, completion-latency distribution, tool-call argument distributions) refreshed from the previous 30-day window; anomaly threshold auto-tunes to maintain target false-positive rate.
- Each detection in the set has a last-tuned date and a false-positive rate; detections that have not fired a true positive in 90 days or that exceed a 20% false-positive rate are reviewed for retirement at the quarterly cycle.

#### Outcome Metrics (L2)

| Metric | Baseline | L2 Target | Source |
|---|---|---|---|
| % Critical-tier artifacts with full prompt/completion + tool-call corpora retained at longest regulatory window | measure | 100% | Log-store retention audit × SM inventory |
| % Critical/High-tier artifacts with anomaly-detection baselines established | measure | ≥90% | Detection telemetry |
| Cross-artifact correlation rules live and firing within last 90 days (or no applicable events in the window) | measure | ≥3 rules active | SIEM rule registry |
| Detection set quarterly update cycle executed (new detections or retirements from IM/ST feedback) | measure | 4 / year | Detection change log |
| Anomaly-detection FP rate for Critical-tier (trend) | measure | actively tuned, trending down | Alert telemetry |
| Compliance evidence bundle ML logging-baseline freshness (Critical-tier) | measure | ≤30 days | Evidence registry |

#### Process Metrics (leading)

- Anomaly baseline refresh cadence honored (monthly for Critical; quarterly for High).
- SIEM correlation rule health monitored (rules producing alerts within expected frequency; no rule silent for >90 days without investigation).
- Detection-gap review calendar on schedule; IM and ST feedback queues reviewed at each quarterly cycle.
- Retention-tier calibration reconciled with SM inventory tier changes, when an artifact is re-tiered, logging depth updated within 14 days (Critical re-tier) or 30 days (other tiers).

#### Effectiveness Metrics (business value)

- True-positive rate improvement quarter-over-quarter as detection tuning loop matures.
- Cross-artifact correlation incidents that would have appeared as isolated per-artifact findings at L1 now unified, reduces IM mean-time-to-understand (MTTU) for complex incidents.
- Compliance evidence bundle ML validation element completing inside PC-Software staleness threshold with no manual intervention.

#### Success Criteria

- Tier-calibrated logging depth applied to 100% of SM inventory with current tier assignments; Critical-tier full corpus retention confirmed.
- SIEM integration live; ≥3 cross-artifact correlation rules active.
- Quarterly detection tuning loop operating with IM-Software and ST-Software feedback; ≥1 net change per cycle (new, updated, or retired detection).
- ≥90% of Critical/High-tier artifacts with anomaly-detection baselines; FP rate tracked and trending down.
- ML logging-baseline validation element fresh (≤30 days) for all Critical-tier artifacts in PC-Software compliance evidence bundles.

---

## Maturity Level 3
### Objective: Express detections as code with automated deployment; apply ML-driven anomaly detection to prompt/completion and tool-call corpora; contribute anonymized detection signatures and telemetry schemas to OWASP LLM, MITRE ATLAS, and sector ISACs

At this level, detections are version-controlled software artifacts deployed through CI/CD, not ad hoc SIEM queries. Anomaly detection on the full prompt/completion and tool-call corpora surfaces novel attack patterns that rule-based detections miss. The detection library and telemetry schemas are contributed back to the AI-assurance ecosystem, OWASP LLM/Agentic Top 10, MITRE ATLAS (`AML.TaXXXX.TXXXX`), OpenTelemetry AI workgroup, and sector ISACs.

#### Dependencies

- **ML-Software L2 (required):** tier-calibrated logging, SIEM integration, and detection tuning loop must be mature before automation is trustworthy.
- **PC-Software L3 (required):** continuous compliance attestation pipeline consumes ML-Software log signals; attestation SLO depends on ML L3 log freshness guarantees.
- **SM-Software L3 (required):** automated inventory and tier-maintenance events trigger automated detection-set updates (new artifact at Critical tier auto-provisions the full detection set for that archetype).

#### Desired Outcomes

- Detection-as-code: every detection in the set is a version-controlled artifact deployed via the same CI/CD pipeline as the AI/HAI software it monitors; a merge request to a detection changes it in production.
- ML-driven anomaly detection on prompt/completion and tool-call corpora identifies attacker-session patterns, unusual completion-distribution shifts, and novel tool-call argument sequences that elude rule-based detections.
- The organization is a net contributor to AI-assurance telemetry standards: OpenTelemetry AI semantic conventions, MITRE ATLAS detection mitigations, and sector ISAC detection-sharing feeds.
- Industry peers can adopt contributed detection schemas and telemetry standards without significant adaptation.

#### Activities

**A) Detection-as-code**

- Every detection in the detection set expressed as a versioned, tested artifact in source control (detection query + metadata: owner, SLA, ATLAS-tactic tag, HAI-TTP tag, false-positive threshold, last-test-result).
- Detection CI/CD pipeline: changes to detection code trigger a test suite (unit tests over synthetic log data, integration tests against a log replay environment) before production deployment.
- Detection deployment via the same change-management pipeline as AI/HAI software; detection changes are reviewed, not applied ad hoc in the SIEM console.
- Detection coverage automatically checked on SM inventory change events: when a new archetype is registered or an artifact is re-tiered to Critical, the automation verifies the required detection set is active for that artifact and opens a gap finding if not.

**B) ML-driven anomaly detection on AI/HAI corpora**

- Apply unsupervised and semi-supervised anomaly models to the prompt/completion and tool-call corpora for Critical and High-tier artifacts:
  - Prompt-sequence anomaly: sessions whose prompt-sequence distribution is a statistical outlier from normal user sessions (attacker-probing signatures, jailbreak-attempt escalation patterns, multi-turn goal-hijack sequences).
  - Completion-distribution anomaly: completions whose embedding distribution shifts from baseline on a rolling window (potential output integrity regression or prompt-injection influence).
  - Tool-call argument anomaly: argument combinations that have never appeared in normal sessions and fall outside the declared tool-scope range (novel TM TTP variant).
- Anomaly model outputs feed the same detection-to-IM-ticket pipeline as rule-based detections; anomaly severity is tagged to the artifact's tier.
- Anomaly models retrained monthly; model retraining produces a new version in the ML-Software model registry with the same lineage tracking as production AI/HAI software.

**C) Contribute detection signatures and telemetry schemas**

- **OpenTelemetry AI workgroup**, contribute semantic conventions for AI/HAI event types (prompt/completion spans, tool-call spans, agent-session traces, training-job events); schema-spec format compatible with the OTel specification process.
- **MITRE ATLAS detection mitigations**, for each detection in the set that corresponds to an ATLAS tactic / technique, propose or validate an `AML.M00xx` mitigation entry (detection-based mitigation type); priority: TA0001 Reconnaissance (shadow-AI emergence), TA0008 Defense Evasion (HITL-gate bypass), TA0013 Exfiltration (canary detection), TA0014 Impact (kill-switch coverage).
- **OWASP LLM / Agentic Top 10**, contribute detection-pattern examples from production telemetry to OWASP LLM Top 10 or Agentic Top 10 community contributions; target at least one detection pattern per cycle.
- **Sector ISACs (FS-ISAC, H-ISAC, IT-ISAC AI working groups)**, share anonymized, generalized detection signatures; target ≥12 signatures per year; signatures must be implementable by partner organizations without significant adaptation.
- Target: ≥2 telemetry-standard contributions per year and ≥12 ISAC detection signatures per year; all contributions anonymized, legally vetted, and maintained, not point-in-time submissions.

#### Outcome Metrics (L3)

| Metric | Baseline | L3 Target | Source |
|---|---|---|---|
| % detections expressed as version-controlled, CI/CD-deployed code artifacts | measure | ≥90% | Detection registry × source control |
| Detection coverage auto-verified on SM inventory change (new/re-tiered artifacts) | measure | 100% within 24h of inventory change | Automation telemetry |
| % Critical/High-tier artifacts with ML-driven anomaly detection active | measure | ≥90% | Anomaly model registry |
| Anomaly model retraining cadence honored | measure | monthly, on schedule | Model registry |
| Telemetry-standard contributions per year | 0 | ≥2 | Contribution log |
| ISAC detection signatures contributed per year | 0 | ≥12 | Contribution log |
| ATLAS detection-mitigation entries proposed or validated | 0 | ≥2 AML.M00xx entries | ATLAS contribution log |

#### Process Metrics (leading)

- Detection CI/CD pipeline health monitored; failed detection-deployment builds are on-call paged the same as production software failures.
- Anomaly model drift monitored (detection model performance against labeled incident records); retraining SLO met monthly.
- Contribution pipeline active, ≥2 telemetry-standard items in-flight (draft, in-review, or submitted) at any time.
- ISAC submission cadence, at least one anonymized detection signature submitted per month.

#### Effectiveness Metrics (business value)

- True-positive incidents surfaced first by ML-driven anomaly detection (vs. rule-based or external notification), trending up quarter-over-quarter.
- Detection deployment lead time (merge to production) measured in hours, not weeks.
- Industry adoption of contributed detection schemas and telemetry conventions tracked; citations, integrations, or standards-body acknowledgments documented.
- ISAC partner organizations citing contributed signatures in their own detection catalogs.

#### Success Criteria

- ≥90% of the detection set expressed as version-controlled, CI/CD-deployed artifacts; detection changes reviewed and deployed through the same change pipeline as AI/HAI software.
- Detection coverage auto-verified for 100% of new or re-tiered SM inventory entries within 24 hours.
- ≥90% of Critical/High-tier artifacts with ML-driven anomaly detection active; anomaly models retrained monthly on schedule.
- ≥2 telemetry-standard contributions per year to OpenTelemetry AI workgroup or equivalent; ≥12 anonymized detection signatures per year to sector ISACs; ≥2 ATLAS `AML.M00xx` mitigation entries proposed or validated.

---

## Key Success Indicators

**Level 1:**
- Per-archetype logging baseline published and instrumented for ≥90% of production AI/HAI software artifacts, covering prompt/completion events, tool-call events, admin-audit events, and identity events per archetype with the fields specified in Activity A.
- ≤12-detection high-signal set live, each tied to a TA-Software archetype threat and HAI TTP / ATLAS tactic tag, with owner, detection query, SLA, and monthly tuning record; false-positive rate tracked per detection.
- Retention meets the longest applicable regulatory window; export path tested annually; EU AI Act Art. 12, GDPR Art. 30, and ISO/IEC 42001 AIMS evidence wiring documented.
- Quarterly deployer-duty drill executed and inside the ≤24-hour SLA; gaps routed to IM-Software.

**Level 2:**
- Tier-calibrated logging depth applied per SM-Software L2 tier-treatment matrix; Critical-tier artifacts retain full prompt/completion and tool-call corpora at the longest regulatory window.
- SIEM integration live with ≥3 cross-artifact correlation rules; ≥90% of Critical/High-tier artifacts with anomaly-detection behavioral baselines and FP rate trending down.
- Quarterly detection tuning loop operating from IM-Software post-incident and ST-Software finding inputs; ≥1 net detection change per cycle.
- ML logging-baseline validation element fresh (≤30 days) for all Critical-tier artifacts in PC-Software compliance evidence bundles.

**Level 3:**
- ≥90% of the detection set expressed as version-controlled, CI/CD-deployed artifacts; detection coverage auto-verified on SM inventory changes within 24 hours.
- ≥90% of Critical/High-tier artifacts with ML-driven anomaly detection active; anomaly models retrained monthly.
- ≥2 telemetry-standard contributions per year to OpenTelemetry AI workgroup or equivalent; ≥12 anonymized detection signatures per year to sector ISACs; ≥2 MITRE ATLAS `AML.M00xx` detection-mitigation entries proposed or validated.

---

## Common Pitfalls

**Level 1:**
- ❌ Logging baseline defined at the archetype level but actual production artifacts never audited against it, gaps accumulate inside the SM inventory without appearing in any backlog.
- ❌ Prompt/completion logging implemented in the application tier but not at the SA-Software reference pattern's designated proxy layer, PII scrubbing is bypassed, retention policy is unenforceable, and content-hash vs. full-text decisions are made inconsistently.
- ❌ Detection set grows without governance because new detections are added at every incident but none are ever retired, the team spends more time triaging false positives than investigating real signals.
- ❌ Tool-call events for agents are logged at the HTTP level (method + status) but not at the argument level, HITL-gate-bypass and tool-scope-violation detections are architecturally impossible without argument-level logging.
- ❌ Deployer-duty evidence view exists as a template document but is never populated for specific artifacts, the quarterly drill is skipped because "we know the logs exist."
- ❌ Retention meets GDPR Art. 30 but not EU AI Act Art. 12 high-risk-system windows, evidence requests for Annex III-adjacent artifacts cannot be satisfied because the relevant log tier was retained for only 30 days.

**Level 2:**
- ❌ Tier-calibrated logging configured at deployment time but not maintained, when an artifact is re-tiered from Medium to Critical, logging depth is not updated; full corpora are absent for the artifact when the first Critical-tier incident fires.
- ❌ SIEM correlation rules are built once and never validated, a correlation rule that has not fired in 90 days may be broken (log format changed, query syntax stale) rather than evidence that no correlatable events occurred.
- ❌ Anomaly baselines established at onboarding and never refreshed, behavioral drift in normal usage makes the baseline stale and FP rates spike over the following quarters.
- ❌ Detection tuning loop exists on paper but IM and ST feedback never actually feeds into the review cycle, the same false-positive detections remain in the set for years because the quarterly process has no dedicated owner.
- ❌ Cross-artifact correlation produces high-severity unified incidents that land in IM-Software without the component-artifact context needed for triage, correlation helps detection but hurts triage unless the correlated event package includes links to the individual artifact findings.

**Level 3:**
- ❌ Detection-as-code pipeline deployed but detection tests use synthetic data that does not resemble production log patterns, tests pass in CI and detections fail silently in production.
- ❌ ML-driven anomaly models retrained on the full log corpus, including attacker-session logs from past incidents, poisoned baseline; the anomaly model learns to treat past attack patterns as normal.
- ❌ Contributed telemetry schemas are published as point-in-time artifacts and then diverge from internal practice, external adopters build against v1.0 while the org operates v1.3 internally; trust erodes.
- ❌ ISAC detection signatures generalized to the point of uselessness, partner organizations cannot implement them without reconstructing the context removed for anonymization.
- ❌ ATLAS `AML.M00xx` contribution targets treated as compliance checkboxes, entries proposed but never followed through to publication because internal legal review creates indefinite delay.

---

## Practice Maturity Questions

**Level 1:**
1. Has a per-archetype logging baseline been published specifying the minimum event schema, fields, retention window, and export path for each AI/HAI software archetype in the SM-Software inventory (LLM-integrated app, agent, RAG, fine-tune/training, eval harness, model-serving service, classical ML), and has compliance of each production artifact been measured against it within the last quarter?
2. Is a high-signal detection set of ≤12 detections active, each with a named owner, detection query, SLA, ATLAS-tactic or HAI-TTP tag, and last-tuned date, including AGH detection, prompt-injection success, tool-scope-violation, training-data-leakage canary, shadow-AI emergence, kill-switch-not-triggered, and HITL-gate-bypass, with false-positive rates tracked per detection and monthly tuning reviews occurring?
3. Has the evidence trail for EU AI Act Art. 12, GDPR Art. 30, and ISO/IEC 42001 AIMS been wired to the ML-Software log store, and has a quarterly deployer-duty drill confirmed that the evidence package for a randomly selected production artifact can be assembled within the ≤24-hour SLA?

**Level 2:**
1. Is tier-calibrated logging depth applied per the SM-Software L2 tier-treatment matrix, Critical-tier artifacts retaining full prompt/completion and tool-call corpora at the longest regulatory window, Low-tier artifacts receiving baseline only, and is this calibration automatically updated when an artifact is re-tiered?
2. Is the SIEM ingesting ML-Software log feeds with ≥3 cross-artifact correlation rules active (covering at minimum multi-artifact AGH, training-to-inference leakage chain, and shadow-AI emergence plus identity pivot), and is a quarterly detection tuning cycle operating from IM-Software post-incident and ST-Software finding inputs?
3. Are ≥90% of Critical/High-tier artifacts running anomaly-detection baselines with behavioral profiles refreshed monthly and FP rates tracked and trending down, and is the ML logging-baseline validation element completing inside the ≤30-day staleness threshold for all Critical-tier artifacts in PC-Software compliance evidence bundles?

**Level 3:**
1. Are ≥90% of detections expressed as version-controlled, CI/CD-deployed code artifacts with automated test coverage against realistic synthetic log data, and is detection coverage auto-verified for 100% of new or re-tiered SM inventory entries within 24 hours of the inventory change event?
2. Are ≥90% of Critical/High-tier artifacts running ML-driven anomaly detection on prompt/completion and tool-call corpora, with anomaly models retrained monthly on production log data, model versions tracked in the ML-Software model registry, and anomaly-model alerts feeding the IM-Software incident backlog through the same detection-to-ticket pipeline as rule-based detections?
3. Has the program contributed ≥2 telemetry-standard artifacts per year to the OpenTelemetry AI workgroup or equivalent and ≥12 anonymized detection signatures per year to sector ISACs, and has it proposed or validated ≥2 MITRE ATLAS `AML.M00xx` detection-mitigation entries, with contributions maintained current and external adoption tracked?

---

**Document Version:** HAIAMM v3.0
**Practice:** Monitoring & Logging (ML)
**Domain:** Software
**Last Updated:** 2026-05-13
**Author:** Verifhai