Monitoring & Logging (ML)

Software Domain - HAIAMM v3.0

Practice Overview

Objective: Establish the logging baseline per AI/HAI software archetype, operate a small high-signal detection set targeted at the top threats from TA-Software, and produce the evidence trail that proves EU AI Act deployer duties, GDPR processor obligations, and ISO/IEC 42001 AIMS requirements, on demand, inside a published SLA.

Description: ML-Software captures the signals produced by every AI/HAI software artifact the organization builds, LLM-integrated applications, autonomous agents, RAG pipelines, fine-tuning and training workloads, evaluation harnesses, model-serving services, and classical ML models in product surfaces. For each archetype it specifies the exact events to capture (prompt/completion, tool-call, admin-audit, identity), the retention window required to satisfy the longest applicable regulation, and the export path that supports auditor review. On top of the logging baseline it operates a bounded, purposeful detection set, each detection tied to a threat from TA-Software, with a named owner, a defined query, and an active tuning record. The full corpus produced by ML-Software is the primary evidence artifact for PC-Software's compliance map: EU AI Act Art. 12 deployer-duty logs, GDPR Art. 30 records of processing, and ISO/IEC 42001 AIMS operational evidence.

Context: Logging AI/HAI software is not the same as logging classic web services. A prompt/completion event requires request-id correlation, model version, classification label, and latency alongside text content or a hash. A tool-call event from an agent must carry the tool name, arguments, return value, principal, and success/fail status, not just an HTTP status code. Training-job events must capture data-source lineage and eval-gate results to support model-promotion audit. None of this exists by default in standard APM or SIEM tooling unless someone has explicitly instrumented the archetype's event schema. ML-Software makes that schema explicit, per archetype, from day one, so the organization is not reconstructing an evidence trail from incomplete telemetry the first time a regulator or incident demands it.

Maturity Level 1

Objective: Establish the per-archetype logging baseline, operate a small high-signal detection set targeting the top TA-Software threats, and produce an on-demand evidence trail that satisfies EU AI Act Art. 12, GDPR Art. 30, and ISO/IEC 42001 AIMS requirements within a published SLA

At this level, the program makes every production AI/HAI software artifact observable with a defined minimum event schema, closes the most dangerous detection gaps (AGH, prompt-injection success, tool-scope violations, shadow-AI emergence), and demonstrates that the resulting log corpus can produce deployer-duty evidence for auditors and regulators on demand.

Dependencies

• SM-Software L1 (required): the inventory and archetype taxonomy define what must be instrumented; ML-Software L1 cannot baseline what SM has not yet catalogued.
• SA-Software L1 (required): reference patterns specify where logs originate (API gateway, proxy layer, agent executor, training pipeline, model-serving endpoint); the patterns specify what to log.
• EH-Software L1 (required): hardened identity controls (SSO, service-principal model, secrets vault) produce the identity events that ML captures; without EH instrumentation there are no usable identity-event log sources.
• TA-Software L1 (required): archetype threat library drives detection priority; the high-signal detection set is selected from TA's top threat list, not invented independently.
• Supports / unblocks: IM-Software L1 (detections become the primary input to the issue backlog); PC-Software L1 (ML log retention and export path constitutes the compliance evidence trail the PC compliance map requires).

Desired Outcomes

• Every production AI/HAI software artifact emits the per-archetype minimum event schema; no artifact produces logs that are architecturally insufficient for deployer-duty evidence.
• Retention meets or exceeds the longest applicable compliance window across the active regulatory set; export path demonstrated for each archetype at least annually.
• A small, bounded detection set is live, each detection tied to a named TA-Software archetype threat, each with an owner, a query, a SLA, and a tuning record; false-positive rate tracked per detection.
• EU AI Act Art. 12 high-risk-system log obligations, GDPR Art. 30 records-of-processing entries, and ISO/IEC 42001 AIMS evidence assets can be satisfied from ML-Software log stores within the published SLA (on-demand pull ≤24 hours).
• Shadow-AI emergence in the engineering environment is detectable from ML signals alone (new outbound flows to LLM provider domains from unregistered services surface immediately in the detection set).

Activities

A) Establish the per-archetype logging baseline

Define and instrument the minimum event schema for each archetype in the SM-Software inventory. Each event record includes: request-id / correlation-id, principal (user or service account), timestamp, archetype tag, and the archetype-specific fields below. PII scrubbing applied per SR-Software data-boundary requirements before logging.

LLM-integrated application: - Prompt event: request-id, user/principal, prompt text or hash (hash where regulated data may be present), classification label, retrieval sources (if RAG-backed), model + version, timestamp. - Completion event: request-id, completion text or hash, model + version, input tokens, output tokens, latency ms, error code. - Guardrail-decision event: filter triggered (name), reason, action taken (pass / block / redact), request-id. - Rate-limit / abuse-detection event: principal, rate-rule triggered, action.

AI agent: - All LLM-integrated-app events above. - Tool-call event: tool name, full arguments (or argument hash for sensitive parameters), return value or hash, principal, timestamp, success/fail, latency ms. - Agent-goal vs. action delta event: declared goal, observed tool invocation, delta flag (triggered when action is outside declared goal scope). - HITL-gate event: gate triggered, approver identity, decision (approved / declined), timestamp. - Kill-switch event: trigger source, timestamp, agent session scope affected, invoking principal.

RAG pipeline: - Retrieval event: sources retrieved (document IDs + classification labels), source provenance, tenant-id for multi-tenant deployments, query-id correlated to the prompt event. - Injection-defense decision event: retrieved-content flagged (yes/no), reason, action (include / exclude / redact).

Fine-tuning / model-training workload: - Training-job event: job-id, data-source list with classification labels and consent-basis flags, model-output identifier, final eval-results summary, training-job duration. - Model-promotion event: model-id, eval gates passed (list), approver, timestamp, promoted-to registry location.

Evaluation harness: - Eval-run event: corpus version, model version, result (pass/fail), regression-delta vs. baseline, eval-job-id.

Model-serving service: - Version-deployment event: model version, deployer principal, canary percentage, deployment-id. - Canary-decision event: canary result, promotion decision, approver. - Rollback event: trigger reason, from-version, to-version, invoking principal, timestamp.

Classical ML model: - Inference event: model-id, feature-set version, prediction, confidence score, principal. - Drift-detection event: drift type (data/concept), magnitude vs. baseline, threshold crossed (yes/no), alert routed.

Admin-audit events (all archetypes): - Config changes to the artifact or its infrastructure components. - Secret rotation events for LLM provider API keys. - IAM changes to roles and service principals associated with the artifact. - Model-registry changes (promotion, deprecation, deletion). - Tool-list changes for agent artifacts (tool added, tool removed, scope modified). - Kill-switch arm / disarm events. - On-call hand-off records.

Identity events (cross-archetype): - SSO sign-ins to model registry consoles, LLM provider admin consoles, CI/CD systems, and code repositories. - Service-principal token use against LLM provider APIs (principal, provider, volume, timestamp).

Retention and exportability: - Retention window: meets or exceeds the longest applicable requirement across active regulations (EU AI Act Art. 12 high-risk logs ≥6 months; GDPR Art. 30 retention per data-class and processing purpose; HIPAA PHI where applicable ≥6 years; sector-specific where applicable). Where multiple retention windows apply to a single artifact, the longest governs. - Export path: JSON or structured CSV export from the log store tested at least annually; on-demand pull SLA ≤24 hours for evidence requests from auditors, regulators, or legal hold. - Log integrity: write-once or append-only storage for admin-audit and deployer-duty evidence tiers; access-control separation between application teams and log store administrators.

B) Operate a small high-signal detection set

L1 target: ≤12 detections, each tied to a TA-Software archetype threat and to at least one HAI TTP tag (EA / AGH / TM / RA) or ATLAS tactic (TA0001 Reconnaissance used for early-stage detection; TA0014 Impact addressed in IM). Each detection has: owner, detection query, SLA (time-to-IM-ticket), and last-tuned date. False-positive rate tracked per detection; monthly tuning review.

Core detection set:

• AGH detection (ATLAS TA0009 Collection / AGH TTP), multi-turn agent goal drift: agent goal declared at session start vs. tool invocations across turns; delta above threshold fires detection.
• Prompt-injection success detection (ATLAS TA0013 Exfiltration / AGH TTP), output exfiltration patterns: completion text matching known exfiltration signatures (data-format markers, credential patterns, canary strings) on outbound completions.
• Tool-scope-violation detection (ATLAS TA0007 Privilege Escalation / EA TTP + TM TTP), agent attempted a tool invocation with a tool or argument outside the published allowlist; fires on any tool-call event where tool name is not in the artifact's declared allowlist or argument exceeds scoped range.
• Training-data-leakage canary detection (ATLAS TA0013 Exfiltration / TM TTP), canary string (injected into training corpus at a known position) emitted verbatim in a completion; signals training-data memorization.
• Shadow-AI emergence detection (ATLAS TA0001 Reconnaissance / EA TTP), new outbound network flow to an unsanctioned LLM provider domain originating from a service that does not appear in the SM-Software inventory; fires on first-seen detection per service-domain pair.
• Kill-switch-not-triggered detection, incident-state flag active for an agent artifact and kill-switch event not present in the log within the declared response SLA; escalates immediately to IM-Software.
• HITL-gate-bypass attempt detection (ATLAS TA0008 Defense Evasion / AGH TTP), HITL-gate event shows repeated decline followed by an attempt to invoke the same destructive tool in the same session without a new gate event.

Each detection routes to the IM-Software backlog on fire; median detection-to-ticket time target ≤1 hour for Critical-tier artifacts.

C) Produce and drill the deployer-duty evidence trail

ML-Software is the primary evidence source for PC-Software's compliance map. At L1, wire the log store to the compliance requirements:

• EU AI Act Art. 12 (high-risk-system logging for deployer duties): for every artifact assessed as Annex III high-risk or carrying a customer-facing decision-affecting output, confirm that prompt/completion, tool-call, and admin-audit events are captured and retained at the required window; produce a deployer-duty evidence view (log record + retention attestation + export test result) for each such artifact.
• GDPR Art. 30 (records of processing): for every artifact processing personal data, the prompt/completion log entries with principal identity, data-class tag, and purpose label constitute the records-of-processing operational entries; link the log-store retention policy to the Art. 30 record for each artifact.
• ISO/IEC 42001 AIMS (operational evidence for the AI Management System): training-job events, model-promotion events, eval-run events, and admin-audit events constitute the AIMS operational records; identify gaps and open IM-Software findings for any archetype not yet emitting these events.

Quarterly deployer-duty drill: pull the deployer-duty evidence package for one randomly selected production AI/HAI software artifact per archetype within the published SLA (≤24 hours from request to assembled package). Record drill results; gaps route to IM-Software.

Outcome Metrics (L1)

Process Metrics (leading)

• Archetype-baseline gap list maintained, every production artifact scored against its archetype baseline; gaps on IM-Software backlog with named owner.
• Detection tuning cadence, monthly review per detection; stale or superseded detections retired; new detection candidates from IM post-incident reviews queued.
• Retention / export test cadence, at least annually per archetype, with documented evidence.
• Deployer-duty drill cadence honored, quarterly, covering all archetypes in rotation.

Effectiveness Metrics (business value)

• Incidents detected internally before external notification, trend over quarters.
• Regulator / auditor / legal evidence requests turned around inside the published SLA (≤24 hours).
• Detection-to-prevention conversion, detections that led to a preemptive control change (SA, SR, EH update) rather than only an incident finding.

Success Criteria

• Per-archetype logging baseline published and instrumented for ≥90% of production AI/HAI software artifacts.
• ≤12-detection high-signal set live, each with owner, detection query, SLA, and monthly tuning record; false-positive rate tracked per detection.
• Retention meets the longest applicable regulatory window for every production artifact; export path tested at least annually.
• EU AI Act Art. 12, GDPR Art. 30, and ISO/IEC 42001 AIMS evidence-trail wiring documented; quarterly deployer-duty drill executed inside the ≤24-hour SLA.

Maturity Level 2

Objective: Calibrate logging depth and detection set to the SM-Software L2 risk-tier rubric; integrate with SIEM for cross-artifact correlation; and feed incident-driven and ST-driven detection updates into a continuous tuning loop

At this level, monitoring intensity matches risk tier. Critical-tier artifacts receive full prompt/completion and tool-call log corpora retained for the longest regulatory window; Low-tier artifacts receive the baseline only. The SIEM ingests ML-Software log feeds and executes cross-artifact correlation rules. The detection set evolves continuously from IM-Software post-incident reviews and ST-Software findings rather than remaining static. Anomaly detection on prompt/completion/tool-call corpora supplements the rule-based set for Critical and High-tier artifacts.

Dependencies

• ML-Software L1 (required): per-archetype logging baseline, detection set, and deployer-duty evidence trail must be operational before tier calibration is meaningful.
• SM-Software L2 (required): risk-tier rubric and tier-treatment matrix drive per-tier logging depth and detection priority; without SM L2 tiers the calibration has no substrate.
• EH-Software L2 (required): hardened control surfaces at L2 produce richer identity and SSO signals; ML-Software L2 consumes them for anomaly baselining.
• IM-Software L1+ (required): post-incident reviews from IM feed detection-tuning and new-detection requests into ML's review cycle.
• ST-Software L1+ (required): security test findings (CI corpus failures, red-team outputs) identify gaps in the detection set and drive new detection candidates.
• Supports / unblocks: PC-Software L2 (tier-calibrated compliance evidence bundles auto-assembly requires ML L2 log completeness signals); IM-Software L2 (richer detections feed the tiered incident playbook with higher-fidelity severity signals).

Desired Outcomes

• Logging intensity visibly differentiates by tier: Critical-tier artifacts retain full prompt/completion and tool-call corpora for the longest regulatory window; Low-tier artifacts produce only the baseline schema.
• SIEM integration enables cross-artifact correlation, a single attacker session interacting with multiple AI/HAI artifacts in sequence surfaces as a correlated detection rather than isolated per-artifact alerts.
• The detection set evolves quarterly from a defined, governed feedback loop (IM post-incident reviews + ST findings + external advisory updates from ATLAS, OWASP LLM, AVID) rather than by ad hoc addition.
• Anomaly detection establishes behavioral baselines for Critical and High-tier artifacts; rule-based detections that fire frequently without incident are refined or retired before the L2 cycle closes.

Activities

A) Tier-calibrated logging depth

Apply the SM-Software L2 tier-treatment matrix to logging configuration:

• Critical tier: full prompt text and completion text (not hashes) retained for the longest regulatory window; full tool-call argument and return corpora retained; admin-audit and identity events at maximum fidelity; all detections tuned to the artifact; per-tenant isolation enforced at the log store (Critical-tier artifact logs partitioned from other tier logs).
• High tier: full prompt/completion text retained; tool-call events at full fidelity; standard admin-audit and identity events; core detections active.
• Medium tier: prompt/completion hashes retained for regulatory window; standard admin-audit; shadow-AI emergence + baseline detections active.
• Low tier: baseline logging schema only; shadow-AI emergence detection only.

For every Critical-tier artifact, the ML log store is the primary source for PC-Software's compliance evidence bundle (per PC-Software L2 staleness thresholds: ML logging-baseline validation ≤30 days for Critical).

B) SIEM integration and cross-artifact correlation

• Ingest all tier-appropriate ML-Software log feeds into the SIEM.
• Author and maintain at least three cross-artifact correlation rules at L2:
• Multi-artifact AGH correlation: the same principal appears in HITL-gate-bypass attempts on two or more agent artifacts in the same session window, fires a unified incident.
• Training-to-inference leakage chain: a canary-string detection in a completion correlates to a training-data-leakage canary detection from the same model-id's training-job event, escalates to Critical regardless of artifact tier.
• Shadow-AI emergence + identity pivot: a shadow-AI emergence detection on a service correlates to an unusual SSO sign-in to an LLM provider admin console from the same service-principal in the same time window.
• Cross-artifact correlation alerts route to IM-Software at the tier of the highest-tier artifact involved.

C) Detection tuning loop: IM post-incident and ST feedback

Operate a quarterly detection review cycle: - IM-Software post-incident reviews that touch a logging or detection gap generate a detection-update request (new detection, tuned query, or retired false-positive rule). - ST-Software CI corpus failures (prompt-injection regression, data-egress canary, kill-switch test) that are not caught by the current detection set generate a detection-gap finding routed to ML-Software. - External advisory updates (MITRE ATLAS new techniques, OWASP LLM Top 10 updates, AVID advisories) are assessed quarterly; each applicable update either adds a new detection candidate or updates an existing detection's query. - Monthly anomaly-baseline refresh for Critical and High-tier artifacts: normal behavior baseline (prompt volume, tool-call patterns, completion-latency distribution, tool-call argument distributions) refreshed from the previous 30-day window; anomaly threshold auto-tunes to maintain target false-positive rate. - Each detection in the set has a last-tuned date and a false-positive rate; detections that have not fired a true positive in 90 days or that exceed a 20% false-positive rate are reviewed for retirement at the quarterly cycle.

Outcome Metrics (L2)

Process Metrics (leading)

• Anomaly baseline refresh cadence honored (monthly for Critical; quarterly for High).
• SIEM correlation rule health monitored (rules producing alerts within expected frequency; no rule silent for >90 days without investigation).
• Detection-gap review calendar on schedule; IM and ST feedback queues reviewed at each quarterly cycle.
• Retention-tier calibration reconciled with SM inventory tier changes, when an artifact is re-tiered, logging depth updated within 14 days (Critical re-tier) or 30 days (other tiers).

Effectiveness Metrics (business value)

• True-positive rate improvement quarter-over-quarter as detection tuning loop matures.
• Cross-artifact correlation incidents that would have appeared as isolated per-artifact findings at L1 now unified, reduces IM mean-time-to-understand (MTTU) for complex incidents.
• Compliance evidence bundle ML validation element completing inside PC-Software staleness threshold with no manual intervention.

Success Criteria

• Tier-calibrated logging depth applied to 100% of SM inventory with current tier assignments; Critical-tier full corpus retention confirmed.
• SIEM integration live; ≥3 cross-artifact correlation rules active.
• Quarterly detection tuning loop operating with IM-Software and ST-Software feedback; ≥1 net change per cycle (new, updated, or retired detection).
• ≥90% of Critical/High-tier artifacts with anomaly-detection baselines; FP rate tracked and trending down.
• ML logging-baseline validation element fresh (≤30 days) for all Critical-tier artifacts in PC-Software compliance evidence bundles.

Maturity Level 3

Objective: Express detections as code with automated deployment; apply ML-driven anomaly detection to prompt/completion and tool-call corpora; contribute anonymized detection signatures and telemetry schemas to OWASP LLM, MITRE ATLAS, and sector ISACs

At this level, detections are version-controlled software artifacts deployed through CI/CD, not ad hoc SIEM queries. Anomaly detection on the full prompt/completion and tool-call corpora surfaces novel attack patterns that rule-based detections miss. The detection library and telemetry schemas are contributed back to the AI-assurance ecosystem, OWASP LLM/Agentic Top 10, MITRE ATLAS (AML.TaXXXX.TXXXX), OpenTelemetry AI workgroup, and sector ISACs.

Dependencies

• ML-Software L2 (required): tier-calibrated logging, SIEM integration, and detection tuning loop must be mature before automation is trustworthy.
• PC-Software L3 (required): continuous compliance attestation pipeline consumes ML-Software log signals; attestation SLO depends on ML L3 log freshness guarantees.
• SM-Software L3 (required): automated inventory and tier-maintenance events trigger automated detection-set updates (new artifact at Critical tier auto-provisions the full detection set for that archetype).

Desired Outcomes

• Detection-as-code: every detection in the set is a version-controlled artifact deployed via the same CI/CD pipeline as the AI/HAI software it monitors; a merge request to a detection changes it in production.
• ML-driven anomaly detection on prompt/completion and tool-call corpora identifies attacker-session patterns, unusual completion-distribution shifts, and novel tool-call argument sequences that elude rule-based detections.
• The organization is a net contributor to AI-assurance telemetry standards: OpenTelemetry AI semantic conventions, MITRE ATLAS detection mitigations, and sector ISAC detection-sharing feeds.
• Industry peers can adopt contributed detection schemas and telemetry standards without significant adaptation.

Activities

A) Detection-as-code

• Every detection in the detection set expressed as a versioned, tested artifact in source control (detection query + metadata: owner, SLA, ATLAS-tactic tag, HAI-TTP tag, false-positive threshold, last-test-result).
• Detection CI/CD pipeline: changes to detection code trigger a test suite (unit tests over synthetic log data, integration tests against a log replay environment) before production deployment.
• Detection deployment via the same change-management pipeline as AI/HAI software; detection changes are reviewed, not applied ad hoc in the SIEM console.
• Detection coverage automatically checked on SM inventory change events: when a new archetype is registered or an artifact is re-tiered to Critical, the automation verifies the required detection set is active for that artifact and opens a gap finding if not.

B) ML-driven anomaly detection on AI/HAI corpora

• Apply unsupervised and semi-supervised anomaly models to the prompt/completion and tool-call corpora for Critical and High-tier artifacts:
• Prompt-sequence anomaly: sessions whose prompt-sequence distribution is a statistical outlier from normal user sessions (attacker-probing signatures, jailbreak-attempt escalation patterns, multi-turn goal-hijack sequences).
• Completion-distribution anomaly: completions whose embedding distribution shifts from baseline on a rolling window (potential output integrity regression or prompt-injection influence).
• Tool-call argument anomaly: argument combinations that have never appeared in normal sessions and fall outside the declared tool-scope range (novel TM TTP variant).
• Anomaly model outputs feed the same detection-to-IM-ticket pipeline as rule-based detections; anomaly severity is tagged to the artifact's tier.
• Anomaly models retrained monthly; model retraining produces a new version in the ML-Software model registry with the same lineage tracking as production AI/HAI software.

C) Contribute detection signatures and telemetry schemas

• OpenTelemetry AI workgroup, contribute semantic conventions for AI/HAI event types (prompt/completion spans, tool-call spans, agent-session traces, training-job events); schema-spec format compatible with the OTel specification process.
• MITRE ATLAS detection mitigations, for each detection in the set that corresponds to an ATLAS tactic / technique, propose or validate an AML.M00xx mitigation entry (detection-based mitigation type); priority: TA0001 Reconnaissance (shadow-AI emergence), TA0008 Defense Evasion (HITL-gate bypass), TA0013 Exfiltration (canary detection), TA0014 Impact (kill-switch coverage).
• OWASP LLM / Agentic Top 10, contribute detection-pattern examples from production telemetry to OWASP LLM Top 10 or Agentic Top 10 community contributions; target at least one detection pattern per cycle.
• Sector ISACs (FS-ISAC, H-ISAC, IT-ISAC AI working groups), share anonymized, generalized detection signatures; target ≥12 signatures per year; signatures must be implementable by partner organizations without significant adaptation.
• Target: ≥2 telemetry-standard contributions per year and ≥12 ISAC detection signatures per year; all contributions anonymized, legally vetted, and maintained, not point-in-time submissions.

Outcome Metrics (L3)

Process Metrics (leading)

• Detection CI/CD pipeline health monitored; failed detection-deployment builds are on-call paged the same as production software failures.
• Anomaly model drift monitored (detection model performance against labeled incident records); retraining SLO met monthly.
• Contribution pipeline active, ≥2 telemetry-standard items in-flight (draft, in-review, or submitted) at any time.
• ISAC submission cadence, at least one anonymized detection signature submitted per month.

Effectiveness Metrics (business value)

• True-positive incidents surfaced first by ML-driven anomaly detection (vs. rule-based or external notification), trending up quarter-over-quarter.
• Detection deployment lead time (merge to production) measured in hours, not weeks.
• Industry adoption of contributed detection schemas and telemetry conventions tracked; citations, integrations, or standards-body acknowledgments documented.
• ISAC partner organizations citing contributed signatures in their own detection catalogs.

Success Criteria

• ≥90% of the detection set expressed as version-controlled, CI/CD-deployed artifacts; detection changes reviewed and deployed through the same change pipeline as AI/HAI software.
• Detection coverage auto-verified for 100% of new or re-tiered SM inventory entries within 24 hours.
• ≥90% of Critical/High-tier artifacts with ML-driven anomaly detection active; anomaly models retrained monthly on schedule.
• ≥2 telemetry-standard contributions per year to OpenTelemetry AI workgroup or equivalent; ≥12 anonymized detection signatures per year to sector ISACs; ≥2 ATLAS AML.M00xx mitigation entries proposed or validated.

Key Success Indicators

Level 1: - Per-archetype logging baseline published and instrumented for ≥90% of production AI/HAI software artifacts, covering prompt/completion events, tool-call events, admin-audit events, and identity events per archetype with the fields specified in Activity A. - ≤12-detection high-signal set live, each tied to a TA-Software archetype threat and HAI TTP / ATLAS tactic tag, with owner, detection query, SLA, and monthly tuning record; false-positive rate tracked per detection. - Retention meets the longest applicable regulatory window; export path tested annually; EU AI Act Art. 12, GDPR Art. 30, and ISO/IEC 42001 AIMS evidence wiring documented. - Quarterly deployer-duty drill executed and inside the ≤24-hour SLA; gaps routed to IM-Software.

Level 2: - Tier-calibrated logging depth applied per SM-Software L2 tier-treatment matrix; Critical-tier artifacts retain full prompt/completion and tool-call corpora at the longest regulatory window. - SIEM integration live with ≥3 cross-artifact correlation rules; ≥90% of Critical/High-tier artifacts with anomaly-detection behavioral baselines and FP rate trending down. - Quarterly detection tuning loop operating from IM-Software post-incident and ST-Software finding inputs; ≥1 net detection change per cycle. - ML logging-baseline validation element fresh (≤30 days) for all Critical-tier artifacts in PC-Software compliance evidence bundles.

Level 3: - ≥90% of the detection set expressed as version-controlled, CI/CD-deployed artifacts; detection coverage auto-verified on SM inventory changes within 24 hours. - ≥90% of Critical/High-tier artifacts with ML-driven anomaly detection active; anomaly models retrained monthly. - ≥2 telemetry-standard contributions per year to OpenTelemetry AI workgroup or equivalent; ≥12 anonymized detection signatures per year to sector ISACs; ≥2 MITRE ATLAS AML.M00xx detection-mitigation entries proposed or validated.

Common Pitfalls

Level 1: - ❌ Logging baseline defined at the archetype level but actual production artifacts never audited against it, gaps accumulate inside the SM inventory without appearing in any backlog. - ❌ Prompt/completion logging implemented in the application tier but not at the SA-Software reference pattern's designated proxy layer, PII scrubbing is bypassed, retention policy is unenforceable, and content-hash vs. full-text decisions are made inconsistently. - ❌ Detection set grows without governance because new detections are added at every incident but none are ever retired, the team spends more time triaging false positives than investigating real signals. - ❌ Tool-call events for agents are logged at the HTTP level (method + status) but not at the argument level, HITL-gate-bypass and tool-scope-violation detections are architecturally impossible without argument-level logging. - ❌ Deployer-duty evidence view exists as a template document but is never populated for specific artifacts, the quarterly drill is skipped because "we know the logs exist." - ❌ Retention meets GDPR Art. 30 but not EU AI Act Art. 12 high-risk-system windows, evidence requests for Annex III-adjacent artifacts cannot be satisfied because the relevant log tier was retained for only 30 days.

Level 2: - ❌ Tier-calibrated logging configured at deployment time but not maintained, when an artifact is re-tiered from Medium to Critical, logging depth is not updated; full corpora are absent for the artifact when the first Critical-tier incident fires. - ❌ SIEM correlation rules are built once and never validated, a correlation rule that has not fired in 90 days may be broken (log format changed, query syntax stale) rather than evidence that no correlatable events occurred. - ❌ Anomaly baselines established at onboarding and never refreshed, behavioral drift in normal usage makes the baseline stale and FP rates spike over the following quarters. - ❌ Detection tuning loop exists on paper but IM and ST feedback never actually feeds into the review cycle, the same false-positive detections remain in the set for years because the quarterly process has no dedicated owner. - ❌ Cross-artifact correlation produces high-severity unified incidents that land in IM-Software without the component-artifact context needed for triage, correlation helps detection but hurts triage unless the correlated event package includes links to the individual artifact findings.

Level 3: - ❌ Detection-as-code pipeline deployed but detection tests use synthetic data that does not resemble production log patterns, tests pass in CI and detections fail silently in production. - ❌ ML-driven anomaly models retrained on the full log corpus, including attacker-session logs from past incidents, poisoned baseline; the anomaly model learns to treat past attack patterns as normal. - ❌ Contributed telemetry schemas are published as point-in-time artifacts and then diverge from internal practice, external adopters build against v1.0 while the org operates v1.3 internally; trust erodes. - ❌ ISAC detection signatures generalized to the point of uselessness, partner organizations cannot implement them without reconstructing the context removed for anonymization. - ❌ ATLAS AML.M00xx contribution targets treated as compliance checkboxes, entries proposed but never followed through to publication because internal legal review creates indefinite delay.

Practice Maturity Questions

Level 1: 1. Has a per-archetype logging baseline been published specifying the minimum event schema, fields, retention window, and export path for each AI/HAI software archetype in the SM-Software inventory (LLM-integrated app, agent, RAG, fine-tune/training, eval harness, model-serving service, classical ML), and has compliance of each production artifact been measured against it within the last quarter? 2. Is a high-signal detection set of ≤12 detections active, each with a named owner, detection query, SLA, ATLAS-tactic or HAI-TTP tag, and last-tuned date, including AGH detection, prompt-injection success, tool-scope-violation, training-data-leakage canary, shadow-AI emergence, kill-switch-not-triggered, and HITL-gate-bypass, with false-positive rates tracked per detection and monthly tuning reviews occurring? 3. Has the evidence trail for EU AI Act Art. 12, GDPR Art. 30, and ISO/IEC 42001 AIMS been wired to the ML-Software log store, and has a quarterly deployer-duty drill confirmed that the evidence package for a randomly selected production artifact can be assembled within the ≤24-hour SLA?

Level 2: 1. Is tier-calibrated logging depth applied per the SM-Software L2 tier-treatment matrix, Critical-tier artifacts retaining full prompt/completion and tool-call corpora at the longest regulatory window, Low-tier artifacts receiving baseline only, and is this calibration automatically updated when an artifact is re-tiered? 2. Is the SIEM ingesting ML-Software log feeds with ≥3 cross-artifact correlation rules active (covering at minimum multi-artifact AGH, training-to-inference leakage chain, and shadow-AI emergence plus identity pivot), and is a quarterly detection tuning cycle operating from IM-Software post-incident and ST-Software finding inputs? 3. Are ≥90% of Critical/High-tier artifacts running anomaly-detection baselines with behavioral profiles refreshed monthly and FP rates tracked and trending down, and is the ML logging-baseline validation element completing inside the ≤30-day staleness threshold for all Critical-tier artifacts in PC-Software compliance evidence bundles?

Level 3: 1. Are ≥90% of detections expressed as version-controlled, CI/CD-deployed code artifacts with automated test coverage against realistic synthetic log data, and is detection coverage auto-verified for 100% of new or re-tiered SM inventory entries within 24 hours of the inventory change event? 2. Are ≥90% of Critical/High-tier artifacts running ML-driven anomaly detection on prompt/completion and tool-call corpora, with anomaly models retrained monthly on production log data, model versions tracked in the ML-Software model registry, and anomaly-model alerts feeding the IM-Software incident backlog through the same detection-to-ticket pipeline as rule-based detections? 3. Has the program contributed ≥2 telemetry-standard artifacts per year to the OpenTelemetry AI workgroup or equivalent and ≥12 anonymized detection signatures per year to sector ISACs, and has it proposed or validated ≥2 MITRE ATLAS AML.M00xx detection-mitigation entries, with contributions maintained current and external adoption tracked?

Document Version: HAIAMM v3.0 Practice: Monitoring & Logging (ML) Domain: Software Last Updated: 2026-05-13 Author: Verifhai