Monitoring & Logging (ML)

Data Domain - HAIAMM v3.0

Practice Overview

Objective: Establish the logging baseline per AI/HAI data archetype, operate a small high-signal detection set targeted at the top TA-Data threats, and produce the evidence trail that proves EU AI Act Art. 12 deployer duties, GDPR Art. 30 records-of-processing obligations, and ISO/IEC 42001 AIMS requirements, on demand, inside a published SLA.

Description: ML-Data captures the signals produced by every AI/HAI data asset the organization operates, training corpora, inference input streams, retrieval stores, prompt/completion log corpora, embedding stores, fine-tuning datasets, and evaluation/test sets. For each archetype it specifies the exact events to capture (ingestion, access, classification routing, PII-redaction decisions, retention-expiry, export, poison-detection scan, and admin-audit events), the retention window required to satisfy the longest applicable regulation, and the export path that supports DSAR fulfillment and auditor review within a published SLA. On top of the logging baseline it operates a bounded, purposeful detection set, each detection tied to a TA-Data archetype threat, with a named owner, a defined query, and an active tuning record. The full corpus produced by ML-Data is the primary evidence artifact for PC-Data's compliance map: EU AI Act Art. 12 deployer-duty logs, GDPR Art. 30 records of processing, and ISO/IEC 42001 AIMS operational evidence.

Context: Logging AI/HAI data assets is not the same as logging classic database access. A retrieval event must carry the retrieved document IDs, classification labels, tenant isolation check result, and query principal, not only an HTTP status code. A training-job event must capture the dataset reference and version, consent basis for all subjects in the training corpus, poison-detection scan result, and eval-gate outcome to support model-promotion audit. A prompt/completion log write event must record what was redacted, by which rule, before writing, not after. None of this exists by default in standard data-warehouse or SIEM tooling unless the archetype's event schema has been explicitly defined and instrumented. ML-Data makes that schema explicit, per archetype, from day one, so the organization can produce deployer-duty evidence on demand rather than reconstructing an incomplete telemetry trail at the moment a regulator, DSAR, or incident demands it.

Maturity Level 1

Objective: Establish the per-archetype logging baseline, operate a small high-signal detection set targeting the top TA-Data threats, and produce an on-demand evidence trail that satisfies EU AI Act Art. 12, GDPR Art. 30, and ISO/IEC 42001 AIMS requirements within a published SLA

At this level, the program makes every production AI/HAI data asset observable with a defined minimum event schema, closes the most dangerous detection gaps (classification drift, retrieval extraction, embedding inversion, training-data canary leakage, cross-border flow violations, and consent-withdrawal propagation failures), and demonstrates that the resulting log corpus can produce deployer-duty evidence for regulators, DSARs, and auditors on demand.

Dependencies

• SM-Data L1 (required): the AI/HAI data inventory and archetype taxonomy define what must be instrumented; ML-Data L1 cannot baseline what SM-Data has not yet catalogued.
• SA-Data L1 (required): reference patterns specify where logs originate (data catalog, pipeline service account, storage access layer, classification scanner, prompt-log write path); the patterns specify what to log.
• EH-Data L1 (required): hardened identity controls (SSO, named service-principal model, secrets vault, audit-log separation) produce the access and identity events that ML-Data captures; without EH-Data instrumentation there are no usable identity-event log sources for data assets.
• TA-Data L1 (required): the archetype threat library drives detection priority; the high-signal detection set is selected from TA-Data's top threat list, not invented independently.
• Supports / unblocks: IM-Data L1 (detections become the primary runtime input to the data issue backlog); PC-Data L1 (ML-Data log retention and export path constitutes the compliance evidence trail the PC-Data priority compliance map requires, including GDPR Art. 30, EU AI Act Art. 12, HIPAA Art. 164.312, and ISO/IEC 42001 AIMS operational records).

Desired Outcomes

• Every production AI/HAI data asset emits the per-archetype minimum event schema; no data asset produces logs that are architecturally insufficient for deployer-duty evidence or DSAR fulfillment.
• Retention meets or exceeds the longest applicable compliance window across the active regulatory set; export path demonstrated for each archetype at least annually.
• A small, bounded detection set is live, each detection tied to a named TA-Data archetype threat and to at least one HAI TTP tag (EA / AGH / TM / RA) or ATLAS tactic; each with an owner, a query, an SLA, and a tuning record; false-positive rate tracked per detection.
• EU AI Act Art. 12 high-risk-system log obligations, GDPR Art. 30 records-of-processing entries, and ISO/IEC 42001 AIMS evidence assets can be satisfied from ML-Data log stores within the published SLA (on-demand pull ≤24 hours; DSAR-capable export ≤72 hours).
• Classification-label drift and unclassified-data-flow events are detectable from ML-Data signals alone.

Activities

A) Establish the per-archetype logging baseline

Define and instrument the minimum event schema for each AI/HAI data archetype in the SM-Data inventory. Each event record includes: event-id / correlation-id, principal (user or service account), timestamp, archetype tag, asset-id linked to the SM-Data inventory, and the archetype-specific fields below. PII scrubbing applied per SR-Data data-boundary requirements before logging where logging the raw field would itself create a regulated-data exposure.

Training corpus / fine-tuning dataset: - Ingestion event: source identifier, classification label, consent basis for data subjects in scope, lineage reference, processing-job identity, timestamp. - Poison-detection scan event: scan tool, scan result (clean / flagged), flagged-item count, scan-job-id, timestamp. - Training-job event: job-id, dataset reference and version, consent-basis summary, model-output identifier, timestamp. - Eval-result-as-gate event: eval-job-id, pass/fail, eval criteria applied, gate outcome (promoted / blocked), timestamp. - Model-promotion event linked to this dataset: model-id, dataset version, approver, timestamp.

Inference input stream: - PII-redaction-decision event: what data class was detected (by category, not raw value), which redaction rule applied, action taken (redact / tokenize / pass), request-id, timestamp. - Classification-routing-decision event: input data class assigned, routing destination selected, routing rule applied, request-id, timestamp. - No-train-flag check event: flag present (yes/no), flag value, request-id, timestamp; alert if no-train flag is absent when required by the applicable processing record.

Retrieval store: - Retrieval event: document IDs retrieved, classification labels of retrieved documents, query principal, tenant-id for multi-tenant deployments, query-id, timestamp. - Retrieval-poisoning detection event: flagged document ID, detection method, action taken (included / excluded), query-id, timestamp. - Per-tenant isolation check event: isolation check passed (yes/no), check method, timestamp. - Ingestion event (when new documents are added): document ID, classification label, source, ingestion principal, timestamp.

Prompt/completion log corpus: - Log-write event: what fields were written, what was redacted (data class, redaction rule), retention-policy applied, log-entry-id, timestamp. - Retention-expiry event: log-entry-id, expiry date reached, deletion action taken (deleted / key-destroyed), verification result, timestamp. - Export event: who exported, destination, format, purpose, approval reference (if applicable), timestamp.

Embedding store: - Access event: principal, operation (read / write / bulk-export), embedding-id or query-id, timestamp. - Inversion-defense decision event: inversion-defense check result (passed / flagged), defense method, action taken, query-id, timestamp. - Per-tenant partition check event: partition-check passed (yes/no), check method, timestamp.

Evaluation / test set: - Access event: principal, operation (read / export), asset-id, purpose, timestamp. - Isolation-check event: eval-data-in-training check result (pass / fail, eval data should not appear in the training corpus), check method, timestamp.

Admin-audit events (all archetypes): - Classification-scheme changes (label added, label removed, label renamed). - Retention-policy changes (policy updated, expiry extended or shortened). - Lineage changes (dataset ancestry updated). - Consent-basis changes (consent basis updated, withdrawal recorded). - Key-rotation events for data-asset encryption keys. - Access-policy changes (RBAC role added/removed, service-account permission granted/revoked). - Transfer-mechanism changes (SCC added, adequacy decision updated, BCR registered).

Identity events (cross-archetype): - SSO sign-ins to data catalog, model registry, vector store, prompt-log store, and classification scanner consoles. - Service-principal token use against data storage APIs (principal, asset accessed, operation, volume, timestamp). - JIT access grant and expiry events for sensitive dataset access (requestor, approver, scope, grant time, expiry, session-end).

Retention and exportability: - Retention window: meets or exceeds the longest applicable requirement across active regulations. Reference set: GDPR Art. 30 records-of-processing (5 years typical for the record itself; access logs per data-class and purpose); EU AI Act Art. 12 deployer-duty logs ≥6 months for high-risk systems; HIPAA access logs for PHI ≥6 years; sector-specific where applicable. Where multiple windows apply to a single data asset, the longest governs. - Export path: structured JSON or CSV export from the log store tested at least annually per archetype; on-demand pull SLA ≤24 hours for evidence requests from auditors, regulators, or legal hold; DSAR-capable export SLA ≤72 hours (to support GDPR Art. 15 subject-access requests for data subjects whose data appears in AI/HAI data assets). - Log integrity: write-once or append-only storage for admin-audit and deployer-duty evidence tiers; access-control separation between data-engineering teams and log store administrators; cryptographic log-integrity verification for Critical-tier asset audit logs.

B) Operate a small high-signal detection set

L1 target: ≤12 detections, each tied to a TA-Data archetype threat and to at least one HAI TTP tag (EA / AGH / TM / RA) or ATLAS tactic. Each detection has: owner, detection query, SLA (time-to-IM-Data-ticket), and last-tuned date. False-positive rate tracked per detection; monthly tuning review.

Core detection set:

• Classification-label drift detection, unlabeled or mis-labeled data flow: a data-pipeline event shows a dataset or document with a classification label that is absent, empty, or lower than the label of its parent corpus; fires on any ingestion or routing event where the classification label field is null or conflicts with the declared parent-corpus label. (ATLAS TA0010 Discovery / EA TTP)
• Unclassified-data-flow detection, a data-pipeline service account reads from a dataset with no classification label on file in the SM-Data inventory; fires on any storage-access event where the asset-id does not match a catalogued asset with a classification label. (ATLAS TA0010 Discovery / EA TTP)
• Retrieval extraction attempt detection, anomalous query volume or pattern on a retrieval store: a single principal issues more than N retrieval queries within a rolling time window (where N is ≥3 standard deviations above baseline for that principal); fires immediately and routes to IM-Data. (ATLAS TA0013 Exfiltration / TM TTP)
• Embedding inversion attempt detection, anomalous access pattern on an embedding store: a principal issues a large bulk-read or bulk-export operation on an embedding store that exceeds the declared operational profile for that principal; fires on any bulk-export event exceeding the size threshold or any access by a principal not in the embedding store's declared access list. (ATLAS TA0013 Exfiltration / TM TTP)
• Training-data canary leakage detection, a canary string injected into a training corpus or fine-tuning dataset at a known position is emitted verbatim in a model completion; fires on the completion event via the ML-Software canary-detection mechanism, correlated back to the training-job event linking to the canary-containing dataset version. (ATLAS TA0013 Exfiltration / TM TTP)
• Cross-border flow violation detection, a regulated data asset crosses a regional storage boundary without a documented transfer mechanism on file; fires when a storage-replication event is logged for a residency-controlled asset and the transfer-mechanism registry has no current, active record for the destination region. (ATLAS TA0013 Exfiltration / EA TTP)
• Retention-policy violation detection, a data asset persists past its documented retention expiry date without a deletion event or a documented retention-extension approval; fires on a daily schedule that compares active asset records against retention-expiry dates in the SM-Data inventory. (ATLAS TA0014 Impact / EA TTP)
• No-train flag flipped without IM-Data review, a no-train flag on an inference input stream or prompt/completion log record is changed from "set" to "unset" without a corresponding IM-Data review approval event in the audit trail; fires immediately on any flag-change event without a matching approval reference. (ATLAS TA0008 Defense Evasion / AGH TTP)
• Consent-withdrawal not propagated detection, a data subject's consent withdrawal is recorded in the consent-management system but the affected training corpus or fine-tuning dataset still contains records linked to that subject's data beyond the propagation SLA (default: 30 days from withdrawal); fires on a weekly schedule comparing the consent-withdrawal registry against the active-dataset lineage registry. (ATLAS TA0014 Impact / EA TTP)

Each detection routes to the IM-Data backlog on fire; median detection-to-ticket time target ≤1 hour for Critical-tier data assets.

C) Produce and drill the deployer-duty evidence trail

ML-Data is the primary evidence source for PC-Data's priority compliance map. At L1, wire the log store to the compliance requirements:

• EU AI Act Art. 12 (high-risk-system logging for deployer duties): for every data asset associated with an Annex III high-risk AI system or with a customer-facing decision-affecting output, confirm that ingestion, access, classification-routing, and admin-audit events are captured and retained at ≥6 months; produce a deployer-duty evidence view (log record + retention attestation + export test result) for each such asset.
• GDPR Art. 30 (records of processing): for every data asset processing personal data, the access, PII-redaction-decision, and consent-basis events constitute the records-of-processing operational entries; link the log-store retention policy to the Art. 30 record for each asset; confirm DSAR-capable export path is operational and tested.
• ISO/IEC 42001 AIMS (operational evidence for the AI Management System): training-job events, model-promotion events linked to dataset versions, eval-result-as-gate events, consent-basis-change events, and admin-audit events constitute the AIMS operational records for the data domain; identify gaps and open IM-Data findings for any archetype not yet emitting these events.

Quarterly deployer-duty drill: pull the deployer-duty evidence package for one randomly selected production AI/HAI data asset per archetype within the published SLA (≤24 hours from request to assembled evidence package; ≤72 hours for DSAR-path drill). Record drill results; gaps route to IM-Data.

Outcome Metrics (L1)

Process Metrics (leading)

• Archetype-baseline gap list maintained, every production data asset scored against its archetype baseline; gaps on IM-Data backlog with named owner.
• Detection tuning cadence, monthly review per detection; stale or superseded detections retired; new detection candidates from IM-Data post-incident reviews queued.
• Retention/export test cadence, at least annually per archetype with documented evidence; DSAR export path tested at least semi-annually.
• Deployer-duty drill cadence honored, quarterly, covering all seven archetypes in rotation.

Effectiveness Metrics (business value)

• Data incidents detected internally before external notification, trend over quarters.
• Regulator / auditor / DSAR evidence requests turned around inside the published SLA.
• Detection-to-prevention conversion, detections that led to a preemptive control change (SA-Data, SR-Data, EH-Data update) rather than only an incident finding.

Success Criteria

• Per-archetype logging baseline published and instrumented for ≥90% of production AI/HAI data assets across all seven archetypes.
• ≤12-detection high-signal set live, each with owner, detection query, SLA, ATLAS-tactic or HAI-TTP tag, and monthly tuning record; false-positive rate tracked per detection.
• Retention meets the longest applicable regulatory window for every production data asset; export path tested at least annually; DSAR-capable export path tested at least semi-annually.
• EU AI Act Art. 12, GDPR Art. 30, and ISO/IEC 42001 AIMS evidence-trail wiring documented; quarterly deployer-duty drill executed inside the ≤24-hour (≤72-hour DSAR) SLA.

Maturity Level 2

Objective: Calibrate logging depth and detection set to the SM-Data L2 risk-tier rubric; integrate with SIEM for cross-archetype correlation; and feed incident-driven and ST-Data-driven detection updates into a continuous tuning loop

At this level, monitoring intensity matches risk tier. Critical-tier data assets, training corpora containing regulated data, prompt/completion log corpora subject to GDPR Art. 30 obligations, inference input streams processing PHI at scale, receive full event logging retained for the longest regulatory window. Low-tier assets receive the baseline only. The SIEM ingests ML-Data log feeds and executes cross-archetype correlation rules. The detection set evolves continuously from IM-Data post-incident reviews and ST-Data security-testing findings rather than remaining static. Anomaly detection on retrieval query patterns, embedding access patterns, and training-pipeline events supplements the rule-based set for Critical and High-tier data assets.

Dependencies

• ML-Data L1 (required): per-archetype logging baseline, detection set, and deployer-duty evidence trail must be operational before tier calibration is meaningful.
• SM-Data L2 (required): risk-tier rubric and tier-treatment matrix drive per-tier logging depth and detection priority; without SM-Data L2 tiers the calibration has no substrate.
• EH-Data L2 (required): hardened access controls (HSM-rooted keys, JIT access, content-inspection DLP) at L2 produce richer identity and access signals; ML-Data L2 consumes them for anomaly baselining.
• IM-Data L1+ (required): post-incident reviews from IM-Data feed detection tuning and new-detection requests into ML-Data's review cycle.
• ST-Data L1+ (required): security-testing findings (CI corpus red, red-team outputs) identify gaps in the detection set and drive new detection candidates.
• Supports / unblocks: PC-Data L2 (tier-calibrated compliance evidence bundles require ML-Data L2 log-completeness signals); IM-Data L2 (richer detections feed the tiered incident playbook with higher-fidelity severity signals).

Desired Outcomes

• Logging intensity visibly differentiates by tier: Critical-tier data assets retain full event corpora for the longest regulatory window; Low-tier assets produce only the baseline schema.
• SIEM integration enables cross-archetype correlation, a retrieval extraction attempt correlated with a bulk embedding export from the same principal in the same session surfaces as a unified detection rather than isolated per-asset alerts.
• The detection set evolves quarterly from a defined, governed feedback loop (IM-Data post-incident reviews + ST-Data findings + external advisory updates from ATLAS, AVID, OWASP LLM, GDPR enforcement decisions, DPA enforcement actions).
• Anomaly detection establishes behavioral baselines for Critical and High-tier data assets; rule-based detections that fire frequently without incident are refined or retired before the L2 cycle closes.

Activities

A) Tier-calibrated logging depth

Apply the SM-Data L2 tier-treatment matrix to logging configuration:

• Critical tier: full event logging retained for the longest regulatory window; all admin-audit and identity events at maximum fidelity; all detections tuned to the asset; per-asset log isolation (Critical-tier asset logs partitioned from other tier logs with separate access-control boundary); audit-log cryptographic integrity verification.
• High tier: full event logging retained at the regulatory minimum; admin-audit and identity events at standard fidelity; core detections active.
• Medium tier: event hashes or structured summaries retained for regulatory minimum; standard admin-audit; classification-drift + baseline detections active.
• Low tier: baseline logging schema only; classification-drift detection only.

For every Critical-tier data asset, the ML-Data log store is the primary source for PC-Data's compliance evidence bundle (per PC-Data L2 staleness thresholds: ML-Data logging-baseline validation ≤30 days for Critical-tier).

B) SIEM integration and cross-archetype correlation

• Ingest all tier-appropriate ML-Data log feeds into the SIEM.
• Author and maintain at least three cross-archetype correlation rules at L2:
• Retrieval-to-embedding exfiltration chain: the same principal issues anomalous retrieval queries on a retrieval store and a bulk embedding export from the associated embedding store within the same session window, fires a unified high-severity detection.
• Training-data canary + consent-withdrawal non-propagation: a canary-leakage detection on a model completion correlates to a consent-withdrawal record for a subject whose data appears in the training corpus version linked to the affected training-job event, escalates to Critical regardless of asset tier, triggers GDPR Art. 33 evaluation.
• Cross-border flow + classification escalation: a cross-border flow violation detection on a regulated data asset correlates with a classification-label change on that asset within the same 24-hour window, combined signal indicating potential deliberate mis-labeling to enable cross-border exfiltration; escalates to Critical.
• Cross-archetype correlation alerts route to IM-Data at the tier of the highest-tier asset involved.

C) Detection tuning loop: IM-Data post-incident and ST-Data feedback

Operate a quarterly detection review cycle: - IM-Data post-incident reviews that touch a logging or detection gap generate a detection-update request (new detection, tuned query, or retired false-positive rule). - ST-Data CI corpus failures (canary detection test, retrieval-poisoning test, embedding-isolation test) that are not caught by the current detection set generate a detection-gap finding routed to ML-Data. - External advisory updates (MITRE ATLAS new techniques, AVID advisories, GDPR enforcement decisions from DPAs that reveal data-processing violations in AI systems, sector-regulator AI advisories) are assessed quarterly; each applicable update either adds a new detection candidate or updates an existing detection's query. - Monthly anomaly-baseline refresh for Critical and High-tier data assets: normal behavior baseline (retrieval query volume per principal, embedding access patterns, ingestion event rates, consent-withdrawal propagation latency) refreshed from the previous 30-day window; anomaly threshold auto-tunes to maintain target false-positive rate. - Each detection in the set has a last-tuned date and a false-positive rate; detections that have not fired a true positive in 90 days or that exceed a 20% false-positive rate are reviewed for retirement at the quarterly cycle.

Outcome Metrics (L2)

Process Metrics (leading)

• Anomaly baseline refresh cadence honored (monthly for Critical; quarterly for High).
• SIEM correlation rule health monitored (rules producing alerts within expected frequency; no rule silent for >90 days without investigation).
• Detection-gap review calendar on schedule; IM-Data and ST-Data feedback queues reviewed at each quarterly cycle.
• Retention-tier calibration reconciled with SM-Data inventory tier changes, when a data asset is re-tiered, logging depth updated within 14 days (Critical re-tier) or 30 days (other tiers).
• DPA enforcement advisory review cadence, quarterly; applicable enforcement decisions assessed for new detection candidates.

Effectiveness Metrics (business value)

• True-positive rate improvement quarter-over-quarter as detection tuning loop matures.
• Cross-archetype correlation incidents that would have appeared as isolated per-asset findings at L1 now unified, reduces IM-Data mean-time-to-understand (MTTU) for complex data incidents.
• Compliance evidence bundle ML-Data validation element completing inside PC-Data staleness threshold with no manual intervention.
• DSAR fulfillment time inside the ≤72-hour SLA with no manual log reconstruction.

Success Criteria

• Tier-calibrated logging depth applied to 100% of SM-Data inventory with current tier assignments; Critical-tier full event corpus retention confirmed.
• SIEM integration live; ≥3 cross-archetype correlation rules active.
• Quarterly detection tuning loop operating with IM-Data and ST-Data feedback; ≥1 net change per cycle (new, updated, or retired detection).
• ≥90% of Critical/High-tier data assets with anomaly-detection baselines; false-positive rate tracked and trending down.
• ML-Data logging-baseline validation element fresh (≤30 days) for all Critical-tier data assets in PC-Data compliance evidence bundles.

Maturity Level 3

Objective: Express detections as code with automated deployment; apply anomaly detection to retrieval, inference, and training event corpora; and contribute anonymized detection signatures and telemetry schemas to OWASP LLM data-detection patterns, sector ISACs, and DAMA

At this level, detections are version-controlled software artifacts deployed through CI/CD, not ad hoc SIEM queries. Anomaly detection on the full retrieval query, embedding access, and training-pipeline event corpora surfaces novel attack patterns that rule-based detections miss. The detection library and telemetry schemas are contributed back to the AI-assurance and data-management ecosystem, OWASP LLM/Agentic Top 10 data-detection patterns, MITRE ATLAS, DAMA, sector ISACs.

Dependencies

• ML-Data L2 (required): tier-calibrated logging, SIEM integration, and detection tuning loop must be mature before automation is trustworthy.
• PC-Data L3 (required): continuous compliance attestation pipeline consumes ML-Data log signals; attestation SLO depends on ML-Data L3 log-freshness guarantees.
• SM-Data L3 (required): automated inventory and tier-maintenance events trigger automated detection-set updates (new data asset at Critical tier auto-provisions the full detection set for that archetype).

Desired Outcomes

• Detection-as-code: every detection in the set is a version-controlled artifact deployed via the same CI/CD pipeline used for data-pipeline code changes; a merge request to a detection changes it in production.
• Anomaly detection on retrieval query patterns, embedding access corpora, and training-pipeline event streams identifies exfiltration attempts, consent-withdrawal non-propagation patterns, and novel data-handling abuse sequences that elude rule-based detections.
• The organization is a net contributor to AI-assurance data-telemetry standards: OWASP LLM data-detection patterns, MITRE ATLAS detection mitigations for data-domain techniques, DAMA data-security logging standards, and sector ISAC detection-sharing feeds.
• Industry peers can adopt contributed detection schemas and telemetry standards without significant adaptation.

Activities

A) Detection-as-code

• Every detection in the detection set expressed as a versioned, tested artifact in source control (detection query + metadata: owner, SLA, ATLAS-tactic tag, HAI-TTP tag, false-positive threshold, last-test-result, archetype tags).
• Detection CI/CD pipeline: changes to detection code trigger a test suite (unit tests over synthetic log data representing realistic data-asset event patterns, integration tests against a log replay environment populated with anonymized historical events) before production deployment.
• Detection deployment via the same change-management pipeline as data-pipeline code; detection changes reviewed and deployed, not applied ad hoc in the SIEM console.
• Detection coverage automatically checked on SM-Data inventory change events: when a new data asset is registered or an existing asset is re-tiered to Critical, the automation verifies the required detection set is active for that archetype and tier, and opens a gap finding if not, within 24 hours.

B) Anomaly detection on AI/HAI data corpora

Apply unsupervised and semi-supervised anomaly models to the event corpora for Critical and High-tier data assets:

• Retrieval query pattern anomaly: retrieval event sequences from a principal that are a statistical outlier from normal query patterns (attacker-probing signatures, bulk retrieval across document classes, multi-session high-volume extraction patterns that individually fall below the rule-based threshold but collectively exceed baseline).
• Embedding access corpus anomaly: embedding store access patterns whose distribution shifts from baseline on a rolling window (bulk export preparation sequences, unusual access to embedding partitions not in the principal's declared operational scope, novel argument patterns in embedding API calls).
• Training-pipeline event stream anomaly: training job events whose dataset composition, consent-basis distribution, or provenance lineage differs statistically from prior approved training runs for the same model family (potential unauthorized data inclusion or consent-basis manipulation).
• Consent-withdrawal propagation latency anomaly: propagation delay between consent-withdrawal events in the consent registry and the appearance of deletion/exclusion events in the training dataset audit log, modeled by principal and dataset; outliers above the distribution trigger early warning before the rule-based retention-violation detection fires.

Anomaly model outputs feed the same detection-to-IM-Data-ticket pipeline as rule-based detections. Anomaly models retrained monthly; model retraining produces a new version tracked in the model registry with the same lineage attestation required for production AI/HAI software.

C) Contribute detection signatures and telemetry schemas

• OWASP LLM / Agentic Top 10 data-detection patterns, contribute data-domain detection pattern examples (retrieval extraction, embedding inversion, training-data canary leakage, consent-withdrawal non-propagation) to the OWASP LLM Top 10 or Agentic Top 10 community; target at least one data-domain detection pattern contribution per release cycle.
• MITRE ATLAS detection mitigations, for each detection in the set that corresponds to an ATLAS tactic / technique, propose or validate an AML.M00xx mitigation entry (detection-based mitigation type); priority: TA0013 Exfiltration (retrieval extraction, embedding inversion, canary leakage), TA0014 Impact (consent-withdrawal, retention violation, no-train-flag bypass).
• DAMA (Data Management Body of Knowledge), contribute data-security logging schema standards for AI/HAI data assets: event schema definitions for training-corpus ingestion events, retrieval-store access events, embedding-store access events, and consent-propagation events; suitable for inclusion in DAMA DMBOK AI data governance chapter.
• Sector ISACs (FS-ISAC, H-ISAC, IT-ISAC AI working groups), share anonymized, generalized detection signatures for AI data-domain threats; target ≥12 signatures per year; signatures must be implementable by partner organizations without significant reconstruction.
• Target: ≥2 telemetry-standard contributions per year and ≥12 ISAC detection signatures per year; all contributions anonymized, legally vetted, and maintained, not point-in-time submissions.

Outcome Metrics (L3)

Process Metrics (leading)

• Detection CI/CD pipeline health monitored; failed detection-deployment builds paged to the on-call data-security engineer the same as production data-pipeline failures.
• Anomaly model drift monitored (detection model performance against labeled incident records); retraining SLO met monthly.
• Contribution pipeline active, ≥2 telemetry-standard items in-flight (draft, in-review, or submitted) at any time.
• ISAC submission cadence, at least one anonymized data-domain detection signature submitted per month.

Effectiveness Metrics (business value)

• True-positive incidents surfaced first by anomaly detection (vs. rule-based or external notification), trending up quarter-over-quarter.
• Detection deployment lead time (merge to production) measured in hours, not weeks.
• Industry adoption of contributed data-domain detection schemas and telemetry conventions tracked; citations, integrations, or standards-body acknowledgments documented.
• ISAC partner organizations citing contributed data-domain signatures in their own detection catalogs.

Success Criteria

• ≥90% of the detection set expressed as version-controlled, CI/CD-deployed artifacts; detection changes reviewed and deployed through the same change pipeline as data-pipeline code.
• Detection coverage auto-verified for 100% of new or re-tiered SM-Data inventory entries within 24 hours.
• ≥90% of Critical/High-tier data assets with anomaly detection active across retrieval query, embedding access, and training-pipeline event dimensions; anomaly models retrained monthly on schedule.
• ≥2 telemetry-standard contributions per year to OWASP LLM, DAMA, or equivalent; ≥12 anonymized detection signatures per year to sector ISACs; ≥2 ATLAS AML.M00xx mitigation entries proposed or validated.

Key Success Indicators

Level 1: - Per-archetype logging baseline published and instrumented for ≥90% of production AI/HAI data assets, covering ingestion, access, classification-routing, PII-redaction-decision, retention-expiry, export, poison-detection, and admin-audit events per archetype with the fields specified in Activity A. - ≤12-detection high-signal set live, each tied to a TA-Data archetype threat and ATLAS-tactic or HAI-TTP tag, with owner, detection query, SLA, and monthly tuning record; false-positive rate tracked per detection. - Retention meets the longest applicable regulatory window (GDPR Art. 30 / EU AI Act Art. 12 / HIPAA as applicable); export path tested annually; DSAR-capable export path tested semi-annually; EU AI Act Art. 12, GDPR Art. 30, and ISO/IEC 42001 AIMS evidence wiring documented. - Quarterly deployer-duty drill executed inside the ≤24-hour (≤72-hour DSAR-path) SLA; gaps routed to IM-Data.

Level 2: - Tier-calibrated logging depth applied per SM-Data L2 tier-treatment matrix; Critical-tier data assets retain full event corpora at the longest regulatory window. - SIEM integration live with ≥3 cross-archetype correlation rules; ≥90% of Critical/High-tier data assets with anomaly-detection behavioral baselines and false-positive rate trending down. - Quarterly detection tuning loop operating from IM-Data post-incident and ST-Data finding inputs; ≥1 net detection change per cycle. - ML-Data logging-baseline validation element fresh (≤30 days) for all Critical-tier data assets in PC-Data compliance evidence bundles.

Level 3: - ≥90% of the detection set expressed as version-controlled, CI/CD-deployed artifacts; detection coverage auto-verified on SM-Data inventory changes within 24 hours. - ≥90% of Critical/High-tier data assets with anomaly detection active; anomaly models retrained monthly. - ≥2 telemetry-standard contributions per year to OWASP LLM / DAMA or equivalent; ≥12 anonymized detection signatures per year to sector ISACs; ≥2 MITRE ATLAS AML.M00xx detection-mitigation entries proposed or validated.

Common Pitfalls

Level 1: - ❌ Logging baseline defined at the archetype level but actual production data assets never audited against it, gaps accumulate in the SM-Data inventory without appearing in any backlog; the quarterly deployer-duty drill is skipped because "we know the logs exist." - ❌ Retrieval-store access logged at the service-account level (service-account X accessed the retrieval store) but not at the document-ID level, retrieval extraction attempt detections are architecturally impossible because the query content and retrieved document IDs are absent from the event. - ❌ Consent-withdrawal events are recorded in the consent-management system but never linked to the training-corpus lineage events in the ML-Data log store, consent-withdrawal non-propagation detection cannot fire because the two event streams are in separate, unlinked systems. - ❌ Detection set grows without governance: new detections are added at every incident but none are ever retired, the team spends more time triaging false positives than investigating real signals. - ❌ DSAR-capable export path is documented in the compliance map but never tested, when the first DSAR arrives for a data subject whose data appears in a training corpus, the export takes 3 weeks because no one has tested the path. - ❌ Retention meets GDPR Art. 30 but not EU AI Act Art. 12 high-risk-system windows, evidence requests for Annex III-adjacent data assets cannot be satisfied because the relevant log tier was retained for only 30 days.

Level 2: - ❌ Tier-calibrated logging configured at asset registration but not maintained, when a data asset is re-tiered from Medium to Critical, logging depth is not updated; full event corpora are absent when the first Critical-tier data incident fires. - ❌ SIEM correlation rules built once and never validated, a correlation rule silent for 90 days may be broken (event schema changed, query syntax stale) rather than evidence that no correlatable events occurred. - ❌ Anomaly baselines established at onboarding and never refreshed, behavioral drift in normal data-pipeline usage makes the baseline stale and false-positive rates spike over subsequent quarters. - ❌ Detection tuning loop exists on paper but IM-Data and ST-Data feedback never feeds into the review cycle, the same false-positive detections remain for years because the quarterly process has no dedicated owner. - ❌ DPA enforcement advisory review not connected to detection tuning, a GDPR enforcement decision revealing a novel AI data-processing violation pattern in a peer organization is not assessed for detection relevance; the same pattern appears in the org's data assets undetected.

Level 3: - ❌ Detection-as-code pipeline deployed but tests use synthetic data that does not resemble realistic data-asset event patterns, tests pass in CI and detections fail silently in production because the synthetic events lack the field combinations present in real retrieval extraction attempts. - ❌ Anomaly models retrained on the full event corpus including labeled attacker-session logs from past incidents, poisoned baseline; the anomaly model learns to treat past retrieval extraction patterns as normal. - ❌ Contributed DAMA or OWASP data-detection schemas are published as point-in-time artifacts and then diverge from internal practice, external adopters build against v1.0 while the org operates v1.4 internally; trust and reusability erode. - ❌ ISAC detection signatures generalized to the point of uselessness, partner organizations cannot implement them without reconstructing the context removed for anonymization; the contribution is technically present but practically unused.

Practice Maturity Questions

Level 1: 1. Has a per-archetype logging baseline been published specifying the minimum event schema, fields, retention window, and export path for each AI/HAI data archetype in the SM-Data inventory (training corpus, inference input stream, retrieval store, prompt/completion log corpus, embedding store, fine-tuning dataset, evaluation/test set), and has compliance of each production data asset been measured against it within the last quarter? 2. Is a high-signal detection set of ≤12 detections active, each with a named owner, detection query, SLA, ATLAS-tactic or HAI-TTP tag, and last-tuned date, including classification-label drift, unclassified-data-flow, retrieval extraction attempt, embedding inversion attempt, training-data canary leakage, cross-border flow violation, retention-policy violation, no-train-flag flip, and consent-withdrawal non-propagation, with false-positive rates tracked per detection and monthly tuning reviews occurring? 3. Has the evidence trail for EU AI Act Art. 12, GDPR Art. 30, and ISO/IEC 42001 AIMS been wired to the ML-Data log store, and has a quarterly deployer-duty drill confirmed that the evidence package for a randomly selected production data asset can be assembled within the ≤24-hour SLA, and has a DSAR-capable export path test been executed within the last 6 months confirming the ≤72-hour DSAR SLA?

Level 2: 1. Is tier-calibrated logging depth applied per the SM-Data L2 tier-treatment matrix, Critical-tier data assets retaining full event corpora at the longest regulatory window, Low-tier assets receiving baseline only, and is this calibration automatically updated when a data asset is re-tiered, with re-tier-to-Critical updates completing within 14 days? 2. Is the SIEM ingesting ML-Data log feeds with ≥3 cross-archetype correlation rules active (covering at minimum retrieval-to-embedding exfiltration chain, training-data canary plus consent-withdrawal correlation, and cross-border flow plus classification-escalation), and is a quarterly detection tuning cycle operating from IM-Data post-incident and ST-Data finding inputs, with DPA enforcement advisory review included in the quarterly cycle? 3. Are ≥90% of Critical/High-tier data assets running anomaly-detection baselines across retrieval query, embedding access, and training-pipeline event dimensions, with behavioral profiles refreshed monthly and false-positive rates tracked and trending down, and is the ML-Data logging-baseline validation element completing inside the ≤30-day staleness threshold for all Critical-tier data assets in PC-Data compliance evidence bundles?

Level 3: 1. Are ≥90% of detections expressed as version-controlled, CI/CD-deployed code artifacts with automated test coverage against realistic synthetic data-asset event patterns, and is detection coverage auto-verified for 100% of new or re-tiered SM-Data inventory entries within 24 hours of the inventory change event? 2. Are ≥90% of Critical/High-tier data assets running anomaly detection on retrieval query, embedding access, and training-pipeline event corpora, with anomaly models retrained monthly on production event data, model versions tracked in the model registry with SLSA-style provenance, and anomaly-model alerts feeding the IM-Data incident backlog through the same detection-to-ticket pipeline as rule-based detections? 3. Has the program contributed ≥2 telemetry-standard artifacts per year to OWASP LLM data-detection patterns, DAMA, or equivalent, and ≥12 anonymized detection signatures per year to sector ISACs, and has it proposed or validated ≥2 MITRE ATLAS AML.M00xx detection-mitigation entries, with contributions maintained current and external adoption tracked?

Document Version: HAIAMM v3.0 Practice: Monitoring & Logging (ML) Domain: Data Last Updated: 2026-05-13 Author: Verifhai