Implementation Review (IR)

Data Domain - HAIAMM v3.0

Practice Overview

Objective: Verify, at go-live and on a recurring cadence, that the actually-deployed data flows feeding AI/HAI systems match the design approved at DR-Data, and that they stay there as pipelines, classification schemes, and consumer AI artifacts evolve.

Description: IR-Data is the configuration and state check for Data-domain assets, the moment a reviewer opens the data catalog, the pipeline metadata, the lineage graph, the encryption configuration, and the vendor admin APIs and confirms that what is deployed matches the DR-Data decision record. At L1 the review runs at go-live, at least annually, and on material change (new data source added, classification scheme change, region change, retention change, new consumer added). At L2, IR-Data consumes catalog-change webhooks, pipeline-metadata events, lineage-graph signals, classification-label-scan deltas, and vendor admin API recurrent probes to detect drift continuously for High and Critical-tier data flows. Findings are severity-tagged and SLA-bound per the SM-Data L2 tier-treatment matrix; they feed IM-Data for tracking and resolution. No-train flags on outbound LLM calls, retention settings, and classification-label propagation are verified recurrently, not trusted from design text alone.

Context: The gap between the approved data-flow design and the running pipeline is the primary source of silent compliance and security exposure in Data-domain AI/HAI programs. A training corpus's classification label is correct in the DR record but a pipeline transformation strips it downstream. An inference-input stream's PII-redaction step is documented in the SA pattern but was bypassed by a hotfix deploy. A vendor LLM's no-train setting is confirmed in the DR record but was reset by the vendor's platform update. A GDPR Art. 17 deletion job is in the design but was disabled when the cron schedule was migrated. IR-Data closes these gaps by making the implementation check systematic, evidence-based, and recurring, not a one-time pre-launch checkbox or a scramble when an incident or a supervisory authority audit reveals a drift.

Maturity Level 1

Objective: Run per-archetype implementation reviews at go-live, annually, and on material change, verifying deployed data flows match the SA-Data pattern, configuration matches the DR-Data decision, and SR-Data REM evidence is current

At this level, the gap between approved design and running data pipeline is systematically checked at the moments it matters most. Every review produces findings with severity tags, named owners, and SLA-bound resolution dates.

Dependencies

• DR-Data L1 (required): the approved DR-Data decision record is the specification IR checks against; without it there is no authoritative baseline.
• SR-Data L1 (required): the REM defines which requirements must be evidenced; IR verifies the evidence is current and accurate.
• SA-Data L1 (required): the SA-Data reference pattern defines the intended configuration shape; IR checks adherence to the pattern's controls.
• EG-Data L1 (required): reviewers must understand Data-domain archetypes and know where to look for pattern controls in catalog metadata, pipeline configs, and vendor admin consoles.
• Supports / unblocks: ST-Data L1 (tests run against the verified configuration), EH-Data L1 (hardening acts on IR findings), IM-Data L1 (IR findings become issues in the backlog), ML-Data L1 (logging configuration verified here feeds monitoring).

Desired Outcomes

• The gap between the DR-Data-approved design and the live data pipeline stays small and short-lived.
• Material pipeline and configuration drift is found by the program, not by an external auditor, a data subject complaint, or an incident.
• Every finding has a named owner, a severity tag, and an SLA-bound resolution date; aging findings are visible to the program sponsor.
• Material changes to a production AI/HAI data flow always trigger a review before the change is in production, new data sources, classification changes, region changes, retention changes, and new consumers do not bypass the gate.
• The SR-Data REM is kept current, evidence cited in the REM is verified to still be accurate, not just present.
• No-train flags on vendor LLM calls are verified via vendor admin API state, not trusted from contract text alone.

Activities

A) Publish the per-archetype data implementation review checklist

One checklist per SM-Data data archetype, focused on the configuration and pipeline state points where production reality most commonly drifts from the approved design. Each item is a yes/no with a required evidence artifact (catalog screenshot, pipeline config export, vendor admin API response, scan result, log sample).

Common spine across all seven archetype checklists: - Classification labels propagating correctly, pull a sample of records at each pipeline stage; verify classification labels match the approved taxonomy and are not silently stripped by transformation steps. Evidence: sample scan output with label audit trail. - Lineage as designed, data-catalog lineage graph matches the approved flow design; no new sources or consumers added since the last DR; lineage records are not stale. - Consent-basis / lawful-basis evidence current, the GDPR Art. 6 (or Art. 9) lawful-basis record cited in the SR-Data REM is still active, not expired or withdrawn; for consent-based processing, a sample of consent records is verified. - Retention enforcement actually running, the deletion job or retention policy is active and has executed within the expected window; evidence: job execution log with last successful run timestamp. - Encryption keys in vault, not in code, encryption configuration matches the approved design; key references point to the declared vault, not to environment variables, code, or config files; confirm via secrets scan in the deployed pipeline artifact. - Access-control matches design, access to the data flow at rest and in motion matches the approved access-control model; service accounts verified against declared principals; no additional human or system principals have been granted access since the last DR. - DSAR queries actually return correct subjects, execute a test DSAR query for a canary subject record; verify the response includes all expected data from this archetype's contribution to the DSAR surface and no unexpected subjects.

Archetype-specific additions:

• Training corpus / training dataset: poison-detection scan results current (scan run within 90 days); data-minimization scope matches the approved design (no new data classes in corpus not approved at DR); opt-out records applied (subjects who have opted out are excluded from the corpus, verify via subject-ID sample check).

• Inference input stream: PII-redaction step is active and functioning, inject a canary PII record into the test path; verify the payload reaching the LLM API has the PII redacted; evidence: test execution log. No-train flag confirmed via vendor admin API state (not from contract text alone), see §B. Classification-gated routing verified, inject a regulated-class canary; verify it does not route to a vendor where no-train has not been confirmed.

• Retrieval store: per-tenant retrieval isolation confirmed at query time, send a query from Tenant A that matches a document indexed under Tenant B; verify zero results or a namespacing error is returned; evidence: test execution log. Corpus-source allow-list matches the approved design, no new indexable sources added since the last DR.

• Prompt/completion log corpus: redaction-at-logging confirmed, inject a canary PII value into a test prompt; verify the log record shows the redacted form, not the raw PII. Retention-expiry enforcement confirmed, verify log records older than the declared retention window have been purged; evidence: query on log store for records past the retention date. Bulk-export control confirmed, verify that bulk export of the log corpus requires DSAR authorization and is not available to general service accounts.

• Embedding store: inversion-defense confirmed, verify the embedding endpoint requires authentication and does not expose raw vectors to unauthenticated callers; evidence: unauthenticated probe result. Per-tenant partitioning confirmed, verify embeddings from Tenant A cannot be retrieved in a nearest-neighbor query from a Tenant B session.

• Fine-tuning dataset: consent-tracking records linked, for a sample of training records, verify the link to the consent record authorizing use is present and the consent is current. Opt-out enforcement confirmed, verify subjects on the opt-out list are absent from the dataset; evidence: subject-ID intersection check.

• Evaluation / test set: isolation from training confirmed, verify via data-catalog lineage that the eval dataset's source records do not appear in any training pipeline artifact store. Access control confirmed, execution rights restricted to named personnel; verify against current IAM or role assignments.

B) Verify no-train flags via vendor admin API probes

No-train settings on outbound LLM API calls are probed via vendor admin APIs for all archetypes that route data to external LLM providers (inference input stream, retrieval store, prompt/completion log corpus), not trusted from DPA text or one-time admin-console screenshots:

• OpenAI: Organization Settings API, confirm data_controls.training_data_sharing is false for all applicable API keys.
• Anthropic: Organization admin settings API, confirm model training usage terms reflect the no-train commitment.
• Amazon Bedrock: AWS Service Control Policy and Bedrock model invocation logging config, confirm no model fine-tuning on customer data paths.
• Google Vertex AI / Gemini: Google Cloud Organization Policy constraints, confirm no training-data usage opt-in is active.
• Other vendors: equivalent admin API or authenticated endpoint where available; UI-based verification with screenshot evidence as fallback.

Probing cadence at L1: at go-live, at each annual review, and on material change. Delta from the previous probe (any setting changed) opens an IR finding. Probing evidence stored with the IR record.

C) Perform reviews at the right moments and track findings to closure

Three triggers at L1:

• Go-live review: before the data flow enters production (or before a new version goes live), verify the as-deployed pipeline and configuration against the DR-Data-approved design. No production cutover with a blocker finding open.
• Annual review: every active AI/HAI data flow reviewed at least annually; scheduled from the SM-Data inventory (last-IR-date field linked to a review-due alert).
• Material-change review: any of the following triggers an ad-hoc review before the change ships to production: new data source added; classification scheme changed; cross-border routing changed or new region added; retention policy changed; new consumer AI artifact added; vendor LLM provider changed; DSAR-surface changed. The material-change trigger is wired to the same signal sources as SM-Data inventory material-change events.

Every review produces zero or more findings. Each finding carries: - Severity: Critical / High / Medium / Low, calibrated to the SM-Data L2 tier-treatment matrix's IM SLA column; at L1 use a consistent judgment rubric pending SM-Data L2 formalization. - Owner: named data-pipeline engineer or data-steward; not "the data team." - SLA: Critical blocker resolved before production cutover or rollback required; High ≤7 days; Medium ≤30 days; Low ≤90 days or accepted residual. - Evidence: after-fix evidence artifact linked to the finding before closure.

Findings feed IM-Data as issues and loop back to SR-Data where a finding reveals that an REM row's cited evidence was inaccurate, the REM row is updated before the finding is closed.

Outcome Metrics (L1)

Process Metrics (leading)

• Annual review calendar populated from the SM-Data inventory; flows nearing review-due date visible in advance.
• Material-change trigger wired to SM-Data inventory material-change events; reviews queued within 5 business days of a confirmed material change.
• Reviewer backlog aging, no single reviewer more than 3 flows overdue.
• SR-Data REM update loop active, % of IR findings that trigger an REM row update for the affected requirement.

Effectiveness Metrics (business value)

• Drift-caught-early rate, findings closed before they reach an incident, a regulatory inquiry, or an external audit.
• REM accuracy improvement, % of REM rows where the cited evidence was verified current vs. stale; stale-evidence rate trends down as the IR / SR-Data feedback loop operates.
• Avoided-incident stories, documented cases where IR caught a configuration regression before it caused a data event or a compliance violation.

Success Criteria

• Per-archetype IR checklists published, owned, and linked from the SM-Data inventory record and the DR-Data decision record, one per SM-Data archetype.
• Go-live, annual, and material-change review triggers wired to the SM-Data inventory; 100% of new AI/HAI data flows in the last 90 days have a go-live IR record.
• ≥90% of active AI/HAI data flows carry a current-year IR record.
• All Critical / blocker findings resolved before production cutover; High findings closed within 7 days with evidence linked.
• No-train flags verified via vendor admin API for ≥80% of flows with outbound LLM calls.

Maturity Level 2

Objective: Detect data-flow drift continuously for Critical and High-tier flows via catalog webhooks, pipeline-metadata events, lineage-graph signals, classification-scan deltas, and vendor admin API recurrent probes; calibrate IR cadence per SM-Data tier

At this level, implementation review stops being a point-in-time check and becomes a continuous signal for Critical and High-tier data flows. Drift sources are wired to automated detection. No-train flags, retention settings, and classification-label propagation are validated recurrently via catalog and vendor APIs, not trusted from configuration snapshots. IR cadence is explicitly differentiated by SM-Data L2 risk tier.

Dependencies

• IR-Data L1 (required): per-archetype checklists, no-train probe baseline, and findings workflow must be established.
• SM-Data L2 (required): the risk-tier rubric drives IR cadence and depth per the tier-treatment matrix (Critical: go-live + semi-annual + continuous drift; High: go-live + annual + material change; Medium: go-live + annual; Low: go-live).
• SA-Data L2 (required): IaC-encoded patterns establish the "correct" baseline that continuous drift detection measures against.
• Supports / unblocks: ST-Data L2 (tests run against the continuously verified configuration), EH-Data L2, ML-Data L2 (monitoring configuration verified here feeds detections).

Desired Outcomes

• Configuration drift on Critical-tier data flows is detected within days, not months.
• No-train flags, retention-enforcement status, and classification-label propagation are verified recurrently via catalog and vendor APIs, not trusted from configuration snapshots.
• Cross-border-flow drift (transfer-mechanism changes, new jurisdictions added to pipeline) is detected and severity-tagged within the tier-detection SLA.
• IR cadence visibly differentiates by tier: Critical gets semi-annual reviews plus continuous drift detection; Low gets go-live only.
• All Critical-tier IR findings carry severity tags and SLAs matching the SM-Data L2 tier-treatment matrix.

Activities

A) Continuous drift detection from catalog, pipeline, lineage, classification-scan, and vendor admin APIs

Wire the following signal sources to an automated drift-detection pipeline for Critical and High-tier data flows:

• Catalog drift (Atlan / Collibra / DataHub / Unity Catalog change webhooks): classification-label changes, ownership changes, policy-tag changes, new downstream consumers added, each triggers an automated diff against the DR-Data-approved baseline; material deviations open an IR finding automatically.
• Pipeline drift (Airflow / dbt / Fivetran metadata changes): DAG changes, transformation changes, new source additions, schedule changes, connector-version changes, compared against the DR-Data-approved pipeline specification; deviations flagged.
• Lineage drift (lineage-graph changes): new upstream sources or downstream consumers appearing in the lineage graph since the last IR; any lineage edge not present in the approved flow design opens an IR finding.
• Classification drift (Macie / BigID / Purview scan deltas vs. baseline): new data classes discovered in the corpus or stream since the last scan baseline; newly discovered PII or regulated data classes in an archetype not approved for that class open Critical or High findings depending on the data class.
• Vendor admin API recurrent probes (no-train / retention / model-training settings for LLM providers): Monthly (Critical-tier) and quarterly (High-tier) probes via vendor admin APIs (OpenAI / Anthropic / Bedrock / Vertex equivalents); any delta from the previous probe (setting changed, permission widened, retention period extended) opens an IR finding with severity matching the data-class impact.
• Cross-border flow drift (transfer-mechanism changes / new country in pipeline): compare routing metadata against the approved cross-border transfer map; new jurisdictions or changed transfer mechanisms open IR findings.

Detection latency targets: Critical-tier drift detection ≤7 days from change event to finding opened; High-tier ≤30 days.

B) Tier-calibrated IR cadence

Publish and enforce per the SM-Data L2 tier-treatment matrix: - Critical: go-live + semi-annual + material-change-triggered + continuous drift detection. - High: go-live + annual + material-change-triggered. - Medium: go-live + annual. - Low: go-live + re-review on material change.

Every data flow in the SM-Data inventory has a last-IR-date and next-IR-due field; Critical-tier flows with no IR in the last 180 days are escalated to the program sponsor.

Drift findings severity-tagged and SLA-bound per the SM-Data L2 tier-treatment matrix: - Critical-tier drift finding: ≤7 days to resolve or accepted residual with named owner. - High-tier drift finding: ≤14 days. - Medium: ≤30 days. - Low: ≤90 days or accepted residual.

All drift findings feed IM-Data automatically.

C) DSAR-query accuracy recurrent verification

For Critical and High-tier data flows contributing to the DSAR surface, execute recurrent DSAR-query accuracy tests: - Insert a canary subject record with known attributes into the data flow's storage. - Execute a subject-access query via the DSAR fulfillment system. - Verify the canary record is returned with correct attributes and that no other subjects' records are co-mingled. - Verify the canary record is correctly excluded from the DSAR response after a deletion request is processed.

Cadence: Critical-tier, quarterly; High-tier, semi-annual. Failures are High findings (DSAR-surface exposure risk).

Outcome Metrics (L2)

Process Metrics (leading)

• Drift-detection pipeline health monitored, % Critical flows producing a fresh signal in the last 7 days; on-call alert if feed silent for >48 hours.
• Vendor API probing calendar maintained; missed probes tracked as process-metric failures.
• DSAR-query test coverage tracked per Critical/High-tier flow per IR cycle.
• IR backlog tier-aware; Critical-tier findings never wait behind Low-tier queue items.

Effectiveness Metrics (business value)

• Drift caught before ST/ML detections or incidents, trend measured over quarters.
• Reduced audit findings on configuration claims, external auditors (SOC 2, ISO 42001, GDPR supervisory authority) find IR evidence sufficient without supplemental screenshots or interviews.
• Vendor admin API probing reveals vendor-side changes (no-train setting reset by product update, retention period lengthened by plan change) before they become data-handling incidents.

Success Criteria

• ≥90% of Critical-tier data flows under continuous drift detection; median detection latency ≤7 days.
• No-train flags verified via vendor admin APIs for ≥80% of Critical/High-tier flows with outbound LLM calls on a monthly (Critical) and quarterly (High) probing cadence.
• 100% of Critical/High-tier flows contributing to the DSAR surface with DSAR-query accuracy tests on record in the current IR cycle.
• Tier-cadence adherence ≥95%; Critical-tier findings aged per the SM-Data L2 tier-treatment matrix SLAs.

Maturity Level 3

Objective: Daily attestation per Critical-tier data flow, classification labels current, retention SLA met, encryption-key-rotation healthy, no-train probe results green, with drift auto-opening IM-Data tickets and configuration baseline schemas contributed to OpenSSF AI Data, DAMA, and OWASP SAMM AI

At this level, configuration for Critical-tier AI/HAI data flows is not periodically reviewed, it is attested continuously. Every Critical data flow produces a daily attestation signal confirming that classification labels, retention enforcement, encryption-key health, and no-train probe results are within tolerance. Drift opens an IM-Data ticket automatically. Per-archetype data-flow configuration baseline schemas are contributed to OpenSSF AI Data, DAMA International, OWASP SAMM AI extensions, and CSA AI Safety Initiative.

Dependencies

• IR-Data L2 (required): catalog-based monitoring, vendor admin API probing, DSAR-query accuracy testing, and tier-calibrated cadence must be in place.
• SA-Data L3 (required): externalized patterns supply the attestation frame for automated compliance scans.
• ML-Data L2+ (required): runtime signals (classification-label telemetry, lineage-graph telemetry, logging-completeness signals) are evidence sources the attestation pipeline reads.
• SR-Data L3 (alignment): machine-readable REM schema at SR-Data L3 provides the evidence-freshness signals the attestation pipeline validates against.

Desired Outcomes

• Every Critical-tier AI/HAI data flow produces a daily attestation signal, classification labels, retention enforcement, encryption-key health, and no-train probe results are continuously within tolerance.
• Drift automatically opens an IM-Data ticket; the program does not wait for the next scheduled review to act.
• Per-archetype data-flow configuration baseline schemas (what "correct" deployment looks like for each archetype at each tier) are published externally.
• IR reviewer-hours per data flow trend down as attestation absorbs routine checks; reviewers focus on novel configurations and exception escalations.

Activities

A) Daily attestation signal for Critical-tier data flows

Each Critical-tier AI/HAI data flow produces a daily composite attestation signal covering four dimensions:

1. Classification labels current: automated catalog scan confirms classification labels match the approved SM-Data taxonomy and have not been silently modified or stripped by pipeline transformations in the last 24 hours.

2. Retention SLA met: deletion-job execution logs confirm the retention enforcement job ran within the expected window and that no records past the retention date remain in the archetype's storage; evidence freshness window: ≤24 hours.

3. Encryption-key-rotation healthy: key-management system (KMS) confirms the encryption key for the data flow is within its declared rotation schedule and has not been exported or replicated outside the vault.

4. No-train probe results green: vendor admin API probe result from the most recent probe cycle confirms the no-train setting is active for all LLM providers handling data from this flow; evidence freshness window: ≤30 days for Critical-tier.

Deviations in any dimension automatically open an IM-Data ticket; the ticket carries the drift dimension, the specific control that failed tolerance, and a link to the DR-Data decision record.

Attestation artifacts are machine-readable, signed, and stored in the SM-Data inventory record. They are regulator-consumable for GDPR Art. 35 DPIA evidence continuity, EU AI Act Art. 10 data-governance records, and ISO/IEC 42001 AIMS operational records, without manual assembly.

B) Contribute per-archetype data-flow configuration baseline schemas

Publish per-archetype IR configuration baseline schemas, defining what "correct" implementation looks like for each AI/HAI data archetype at each SM-Data tier, to: - OpenSSF AI Data working group, reference attestation schema for AI data-flow implementations; machine-readable format. - DAMA International AI Data Governance, per-archetype configuration controls with evidence-type definitions. - OWASP SAMM AI extensions, Verification function, Implementation Review stream; practitioner-level checklist items. - CSA AI Safety Initiative AI Controls Matrix, per-archetype configuration controls mapped to CSA control categories.

Internal practice remains aligned to the published external versions; internal-only deviations are proposed as upstream changes.

C) Automated drift-to-IM escalation and SLA enforcement

• All IR findings (whether from daily attestation or from periodic reviews) flow into IM-Data automatically with severity and SLA pre-populated from the SM-Data L2 tier-treatment matrix.
• IM-Data SLA clock starts when the finding is opened; overdue Critical findings escalate to the program sponsor automatically at 50% and 100% of the SLA window.
• Post-incident reviews in IM-Data that touch a data-flow configuration or classification control automatically re-examine the IR record for the affected data flow, was the drift detectable earlier? What attestation rule would have caught it? The answer updates the attestation rule and the IR checklist.

Outcome Metrics (L3)

Process Metrics (leading)

• Attestation-pipeline health monitored, % Critical data flows producing a fresh attestation signal in the last 24 hours; on-call paged if any Critical flow silent for >24 hours.
• Schema publication pipeline, at least one schema in-draft, in-review, or published at any time.
• IM-Data escalation automation tested quarterly, confirm Critical-finding auto-escalation fires correctly for a synthetic finding.
• Post-incident IR feedback loop active, % of IM-Data post-incident reviews that produce an attestation rule update.

Effectiveness Metrics (business value)

• IR reviewer-hours per data flow trending down as attestation absorbs routine checks.
• Zero Critical-tier go-live events where the DR-Data-approved design and the deployed configuration are materially different.
• Auditor configuration-claim findings approaching zero as daily attestation corroborates internal records; GDPR supervisory authority inquiries answered via machine-readable attestation artifacts without manual assembly.
• External recognition, configuration baseline schemas cited by peer organizations, security tooling vendors, or regulatory guidance documents.

Success Criteria

• Daily attestation operating for ≥90% of Critical-tier data flows across all four dimensions (classification labels, retention SLA, encryption-key-rotation health, no-train probe results); deviations auto-opening IM-Data tickets within 1 hour.
• Zero stale-evidence violations for Critical-tier REMs; High-tier stale-evidence rate trending down.
• Per-archetype data-flow configuration baseline schemas published to OpenSSF AI Data / DAMA / OWASP SAMM AI / CSA with documented external adoption.
• IR reviewer-hours per Critical data flow per year trending down over two consecutive quarters as attestation absorbs routine checks.

Key Success Indicators

Level 1: - Per-archetype IR checklists published, one per SM-Data archetype (training corpus, inference input stream, retrieval store, prompt/completion log corpus, embedding store, fine-tuning dataset, evaluation/test set), covering classification-label propagation verification, lineage-as-designed verification, consent-basis currency check, retention-enforcement verification, encryption-key-vault binding check, access-control verification, and DSAR-query accuracy test; inference-input-stream checklist includes PII-redaction canary test and no-train vendor admin API probe. - Go-live, annual, and material-change review triggers wired to the SM-Data inventory; 100% of new AI/HAI data flows in the last 90 days have a go-live IR record; ≥90% of active data flows carry a current-year IR record. - All Critical / blocker findings resolved before production cutover; High findings closed within 7 days with evidence linked; findings-aging dashboard reviewed monthly by the program sponsor. - No-train flags verified via vendor admin API for ≥80% of flows with outbound LLM calls; SR-Data REM update loop active.

Level 2: - ≥90% of Critical-tier data flows under continuous drift detection (catalog-change webhooks, pipeline-metadata changes, lineage-graph signals, classification-scan deltas, vendor admin API recurrent probes, cross-border-flow routing changes); median detection latency ≤7 days. - No-train flags and retention settings verified via vendor admin APIs on a monthly (Critical) and quarterly (High) probing cadence, not from DPA text alone, for ≥80% of Critical/High-tier flows with outbound LLM calls. - 100% of Critical/High-tier data flows contributing to the DSAR surface with DSAR-query accuracy tests on record in the current IR cycle. - Tier-cadence adherence ≥95%: Critical on semi-annual + continuous, High on annual, Medium on annual, Low on go-live + material-change.

Level 3: - ≥90% of Critical-tier data flows producing a daily attestation signal across all four dimensions (classification-label currency, retention SLA, encryption-key-rotation health, no-train probe results green); deviations auto-opening IM-Data tickets within 1 hour. - Zero stale-evidence violations for Critical-tier REMs; evidence freshness windows enforced per the attestation pipeline. - Per-archetype data-flow configuration baseline schemas published to OpenSSF AI Data / DAMA / OWASP SAMM AI / CSA with documented adoption. - IR reviewer-hours per Critical data flow per year trending down over two consecutive quarters as attestation absorbs routine checks.

Common Pitfalls

Level 1: - ❌ IR treated as a one-time go-live formality, no annual re-review and no material-change trigger; classification drift, new data sources, and retention-policy changes accumulate silently for quarters until an audit or a subject-access complaint surfaces them. - ❌ Reviewers take the DR-Data decision record at face value without checking the data catalog or pipeline config, the classification label is declared in the checklist but never verified in the live catalog; the label was stripped by a transformation step. - ❌ No-train and retention settings verified from DPA text or one-time admin-console screenshots without opening the vendor admin API, the setting can be reset by a vendor product update and the team does not know. - ❌ DSAR-query accuracy never tested, the IR checklist has a "DSAR surface mapped: yes" box that is checked without executing a canary subject query and verifying the response. - ❌ Finding severity is flat, all findings treated as Medium regardless of impact, so a classification-label gap on a training corpus sits in the same queue as a minor retention-schedule discrepancy. - ❌ Material-change trigger is not wired to SM-Data inventory events, new data sources, classification changes, and cross-border routing changes ship to production without triggering an IR.

Level 2: - ❌ Drift-detection pipeline ingests catalog-change events but generates no findings on deltas, the pipeline exists but automated finding creation was never configured; drift detection is manual in practice. - ❌ Vendor admin API probing is configured once at onboarding and never re-run, a no-train setting reset by a vendor product update is undetected for months. - ❌ Classification-scan deltas are generated but never compared to the DR-Data-approved baseline, the Macie / BigID / Purview scan runs, but the output is never diffed against the approved classification scope; new data classes discovered in the corpus are not flagged. - ❌ Tier-calibrated cadence exists on paper but Critical and Low-tier flows sit in the same review queue with no prioritization, Critical-tier flows wait behind Low-tier backlogs. - ❌ Drift findings from automated detection dead-end in an alert dashboard rather than auto-opening IM-Data tickets, findings age without owners.

Level 3: - ❌ Daily attestation signals show green across all Critical data flows but the underlying checks cover only retention-job run status, classification-label propagation, consent-basis expiry, cross-border routing, and encryption-key-vault binding are not checked; attestation is cosmetic. - ❌ Configuration baseline schemas published externally diverge from internal practice, what is published reflects the L1 checklist; internal practice has advanced to L2 tooling and L3 vendor API probing; external adopters build on a stale baseline. - ❌ Attestation-exception queue overwhelms the team because configuration tolerance thresholds are too tight, every pipeline dependency version bump triggers a deviation; reviewers suppress the signal source rather than tune the tolerance rules. - ❌ Post-incident IR feedback loop exists in policy but never fires in practice, IM-Data post-incident reviews do not include the IR-record re-examination step; attestation rules never update from incident learning.

Practice Maturity Questions

Level 1: 1. Is there a published, per-archetype IR checklist, one per SM-Data archetype (training corpus, inference input stream, retrieval store, prompt/completion log corpus, embedding store, fine-tuning dataset, evaluation/test set), covering classification-label propagation verification, lineage-as-designed verification, consent-basis currency check, retention-enforcement verification, encryption-key-vault binding check, access-control verification, and DSAR-query accuracy test, with the inference-input-stream checklist including a PII-redaction canary test and a no-train vendor admin API probe? 2. Do 100% of new AI/HAI data flows going to production in the last 90 days carry a go-live IR record, and do ≥90% of all active data flows carry a current-year IR record, with material-change triggers wired to SM-Data inventory events, Critical / blocker findings resolved before production, and High findings closed within 7 days with evidence linked? 3. Are findings severity-tagged and tracked in IM-Data with named owners and SLA-bound closure dates, and does every IR finding that reveals stale or inaccurate REM evidence trigger an SR-Data REM row update before the finding is closed?

Level 2: 1. Are ≥90% of Critical-tier AI/HAI data flows under continuous drift detection, via data-catalog change webhooks, pipeline-metadata events, lineage-graph signals, classification-label-scan deltas, vendor admin API recurrent probes, and cross-border-flow routing monitoring, with median detection latency ≤7 days and automated finding creation on material deviations? 2. Are no-train flags and retention settings verified via vendor admin APIs (OpenAI / Anthropic / Bedrock / Vertex / equivalent) on a monthly (Critical) and quarterly (High) probing cadence, not from DPA text alone, covering ≥80% of Critical/High-tier flows with outbound LLM calls, with deltas from the previous probe opening IR findings with severity matching the data-class impact? 3. Are 100% of Critical/High-tier data flows contributing to the DSAR surface covered by DSAR-query accuracy tests in the current IR cycle, confirming canary-subject inclusion, correct attribute return, and post-deletion exclusion, and is tier-cadence adherence ≥95% with Critical-tier findings aged per the SM-Data L2 tier-treatment matrix SLAs?

Level 3: 1. Are ≥90% of Critical-tier AI/HAI data flows producing a daily attestation signal across all four dimensions (classification-label currency, retention SLA, encryption-key-rotation health, no-train probe results green), with deviations auto-opening IM-Data tickets within 1 hour and zero stale-evidence violations for Critical-tier REMs? 2. Has the program published per-archetype data-flow configuration baseline schemas to OpenSSF AI Data, DAMA, OWASP SAMM AI, or CSA AI Safety Initiative, with documented adoption and internal practice aligned to the published versions, and is IR reviewer-hours per Critical data flow per year trending down over two consecutive quarters? 3. Is the post-incident IR feedback loop operational, IM-Data post-incident reviews include a mandatory IR-record re-examination step, and ≥1 attestation rule update is produced per material incident, ensuring incident learning continuously improves attestation coverage?

Document Version: HAIAMM v3.0 Practice: Implementation Review (IR) Domain: Data Last Updated: 2026-05-13 Author: Verifhai