Design Review (DR)

Data Domain - HAIAMM v3.0

Practice Overview

Objective: Operate the design checkpoint between intake approval and build-out for every new data flow feeding AI/HAI systems, confirming the proposed flow follows the applicable SA-Data reference pattern, covers the SR-Data requirements pack, and documents residual risks before pipeline engineering begins.

Description: DR-Data is the single moment where data architecture (SA-Data), requirements (SR-Data), and threats (TA-Data) meet a specific planned data flow. At L1 the review is deliberate but lean: a per-archetype design checklist, a named reviewer, and a written decision (approve / approve-with-conditions / send-back) stored against the SM-Data inventory record. The review runs before the data-pipeline team begins build-out, catching deviations when they cost hours to correct, not sprints. A two-lane model routes Low / Medium-tier data flows to an async fast-lane (≤2 BD) and High / Critical-tier or deviation cases to a full-lane architect review (≤5 BD) that includes DPO and data-steward participation. Loop-back signals ensure the review process improves SA-Data patterns and SR-Data packs over time rather than accumulating silent compliance debt.

Context: Without a data-flow design checkpoint, AI/HAI data pipelines ship without verified consent basis, without documented cross-border transfer mechanisms, without poison-detection scans on training corpora, and without DSAR-surface mapping. The SA reference pattern and SR requirements pack exist, but teams skip them under delivery pressure, deviate without recording rationale, or simply build before the archetype pattern is consulted. DR-Data enforces the handoff between "data intake approved" and "pipeline build begins," making deviations visible and deliberate. GDPR Art. 35 requires a DPIA for high-risk processing of personal data; the DR decision record is the documented pre-build decision that DPIA evidence references. EU AI Act Art. 10 requires data-governance practices for training data used in high-risk AI; the DR record is the traceability artifact.

Maturity Level 1

Objective: Run a per-archetype design checkpoint for every new AI/HAI data flow before pipeline build-out, producing a written decision traceable to the SA-Data reference pattern, SR-Data requirements pack, and TA-Data threat snapshot

At this level, design review is a consistent gate, not a heroic intervention when data quality or compliance failures surface post-deployment. Every new data flow above the triage threshold receives a review before pipeline engineering begins, and every review produces a written decision linked to the SM-Data inventory record.

Dependencies

• SA-Data L1 (required): the reference patterns for each data archetype are what the checklist measures the proposed design against; without patterns there is no review baseline.
• SR-Data L1 (required): the requirements pack (base plus archetype deltas, including the REM) defines the acceptance bar; the REM is the primary evidence input to the review.
• TA-Data L1 (required): the per-archetype threat snapshot names what the design must defend against; the reviewer walks the snapshot's top threats against the proposed flow design.
• EG-Data L1 (required): reviewers must be able to recognize Data-domain archetypes, data-specific HAI TTPs, and ATLAS data-attack tactics before they can produce a credible DR decision.
• Supports / unblocks: IR-Data L1 (implementation reviews check actual deployed flows against the approved design), ST-Data L1 (tests target the approved flow architecture), IM-Data L1 (incident triage references design assumptions).

Desired Outcomes

• Every new AI/HAI data flow above the triage threshold is reviewed before pipeline build-out; no production deployment occurs without a DR decision on file.
• DR decisions are written, versioned, and stored against the SM-Data inventory record, not tribal knowledge held by the reviewing architect or DPO.
• Deviations from SA-Data reference patterns are approved or rejected explicitly with a named reviewer, a rationale, and a residual-risk acceptance.
• DPIA triggers are identified at design time, not discovered post-deployment, and linked to the DR decision record.
• Review is timeboxed: teams know whether to expect a 2 BD async check or a 5 BD architect session based on the SM-Data tier and deviation status.
• Recurring deviation themes feed back into SA-Data (pattern updates) and SR-Data (pack updates), the review accumulates organizational learning, not only per-ticket decisions.

Activities

A) Publish the per-archetype AI/HAI Data Design Checklist

One checklist per SM-Data data archetype, derived from the applicable SA-Data reference pattern and keyed to the SR-Data base pack and archetype delta. Each item is a yes/no with an evidence pointer. The seven checklists share a common spine and carry archetype-specific additions.

Common spine across all seven checklists: - Classification labeling, the data flow's classification label is declared, consistent with the SM-Data taxonomy, and propagated through the pipeline design. - Lineage / provenance, every dataset has a documented origin, last-verified date, and trust classification; lineage is traceable from source to consumer. - Consent basis / lawful basis, the lawful basis under GDPR Art. 6 (or Art. 9 for special-category data) is documented; for personal-data flows, the consent or lawful-basis evidence is linked from the DR record. - Retention policy, retention period declared and technically enforceable; deletion mechanism specified. - Cross-border transfer mechanism, if the flow crosses a GDPR Chapter V boundary or equivalent, the transfer mechanism (SCCs, adequacy decision, BCRs) is named and on file. - Encryption at rest and in transit, encryption algorithm, key management, and vault storage declared; no keys in code or environment variables. - Access-control model, access to the data flow at rest and in motion is restricted to declared principals; service-account model documented. - DSAR surface, the flow's contribution to the organization's DSAR response surface is mapped; subject-level queryability confirmed or documented as a gap with compensating control. - Classification-label propagation, labels applied at ingestion propagate to downstream consumers and are not silently stripped by transformation steps.

Archetype-specific additions:

• Training corpus / training dataset: DPIA trigger assessment (high-risk processing per GDPR Art. 35?); poison-detection scan scheduled pre-training; data minimization scope review (is the corpus scoped to what is necessary for the stated capability?); opt-out-path design (mechanism for data subjects to exclude their data from training).

• Inference input stream: PII-redaction-edge design (where in the pipeline is PII redacted before the payload reaches the LLM provider?); no-train probe target (the no-train flag for the LLM provider is confirmed as a design requirement, not an assumption); classification-gated routing design (regulated input must not route to a vendor where no-train has not been verified).

• Retrieval store: per-tenant isolation design (data from one tenant must not be retrievable by another at query time); injection-defense scope (prompt structure separates retrieval results from instructions, retrieved content treated as untrusted); corpus-source allow-list declared (all indexable sources enumerated; no dynamic source addition without change review).

• Prompt/completion log corpus: redaction-at-logging design (PII redacted before logs are written, not post-hoc); retention-by-archetype policy declared (prompt/completion logs have a shorter retention window than training data); export-control design (logs are not bulk-exportable without DSAR authorization).

• Embedding store: inversion-defense design (embedding endpoint does not expose raw vectors without access control; nearest-neighbor query scope bounded); per-tenant partitioning design (embeddings from different tenants stored in logically or physically separated namespaces).

• Fine-tuning dataset: DPIA trigger assessment; consent-tracking design (mechanism to link each training sample to the consent record that authorizes its use); opt-out-path design (mechanism to exclude opted-out subjects from fine-tuning runs); data minimization scope review.

• Evaluation / test set: isolation-from-training design (eval data does not flow to any training pipeline, isolation boundary is explicit in the architecture diagram); reproducibility design (pinned dataset versions); access control restricted to named personnel.

B) Triage and route reviews by risk tier and deviation status

The two-lane model is driven by the SM-Data tier assignment and the deviation flag:

• Fast-lane (Low / Medium tier, on-pattern, no cross-border, no DPIA trigger): async checklist review by the designated reviewer; target SLA ≤2 business days. Output: one structured decision record, approve / approve-with-conditions (explicit list) / send-back (reasons stated), stored against the SM-Data inventory record.
• Full-lane (High / Critical tier OR any cross-border flow OR regulated data class OR DPIA trigger OR pattern deviation): architect review with the data-pipeline team, DPO, and data-steward walking the SA-Data reference pattern section-by-section; target SLA ≤5 business days. Output: written decision record with the residual-risk list reviewed by a named architect and signed by the DPO for personal-data flows.

Triage rules at L1 (before SM-Data L2 tiers are established): training corpora, fine-tuning datasets, and inference-input streams processing regulated data (PII, PHI, PCI) default to full-lane. Prompt/completion log corpora and embedding stores default to full-lane if multi-tenant. All others default to fast-lane with override to full-lane available on reviewer judgment.

Decision record contents (both lanes): decision (approve / approve-with-conditions / send-back); checklist completed with evidence pointers; deviations listed with rationale; residual risks listed with named owner and expiry; reviewer name and date; DPO acknowledgment for personal-data flows; links to SM-Data inventory record, TA-Data threat snapshot, SR-Data REM, and DPIA if triggered.

C) Close the loop with SA-Data, SR-Data, and IM-Data

Design review is a learning surface for the data-domain program:

• SA-Data pattern update trigger: three deviations in the same direction for the same archetype auto-queue a pattern-update review with SA-Data ownership. Recurring deviations signal the pattern is miscalibrated.
• SR-Data pack update trigger: an SR-Data requirement repeatedly waived with a compensating control auto-queues an SR pack-revision review.
• IM-Data incident feedback loop: every IM-Data incident re-examines the DR decision record that approved the affected data flow. Was the issue visible at design time? Which checklist item would have caught it? The answer updates the checklist.
• DPIA feedback loop: completed DPIAs are linked back to the DR record; DPIA findings that require design changes route back through the checklist before build-out.

Outcome Metrics (L1)

Process Metrics (leading)

• Reviewer population staffed and trained (EG-Data L1 completion confirmed); named lead reviewer per archetype.
• DPO engagement confirmed for all personal-data full-lane reviews; DPO schedule reserved.
• Fast-lane vs. full-lane ratio monitored, a drift toward all-fast-lane may indicate under-review.
• Pattern-deviation rate tracked by archetype, feeds the SA-Data pattern-update trigger.
• Checklists aligned to current SA-Data reference patterns and SR-Data pack; updated within 30 days of any SA or SR change.

Effectiveness Metrics (business value)

• Issues caught at design vs. caught at IR or in incident, design-stage catch rate trends up as the program matures.
• DPIA completion rate before build-out, fewer post-deployment compliance surprises.
• SA-Data/SR-Data update volume driven by DR feedback, a healthy program generates pattern and pack improvements, not only per-ticket decisions.

Success Criteria

• Per-archetype data design checklists published, versioned, and traceable to the applicable SA-Data reference pattern, SR-Data requirements pack, and TA-Data threat snapshot, covering all seven Data-domain archetypes.
• Two-lane review model operational with published SLAs (≤2 BD fast-lane, ≤5 BD full-lane) and named lead reviewers per archetype.
• ≥95% of AI/HAI data flows going to production in the last 90 days carry a completed DR decision record before pipeline build-out begins.
• DPIA trigger identification wired into the checklist; DPO acknowledgment required for all personal-data full-lane reviews.
• SA-Data pattern-update and SR-Data pack-update triggers wired so recurring deviations and waived requirements feed back; every IM-Data incident re-examines the DR record that approved the affected data flow.

Maturity Level 2

Objective: Upgrade Critical-tier data-flow reviews to scenario-based walkthroughs driven by TA-Data per-flow threat models, detect design drift on a published cadence, and run joint DR-Data / DR-Software reviews for Critical-tier data flows feeding first-party AI artifacts

At this level, design review for Critical-tier data flows moves from checklist conformance to scenario-based conversations. The TA-Data per-flow deep threat models (available at TA-Data L2) drive the scenarios using the SM-Data L2 risk-tier rubric. Design drift, the live data flow diverging from the approved design, is detected on a published cadence and automatically re-routed to DR. Where a Critical-tier data flow feeds a first-party AI artifact (a training corpus flowing to a fine-tune workload, an inference-input stream flowing to an LLM-integrated app), DR-Data coordinates a joint review with DR-Software.

Dependencies

• DR-Data L1 (required): per-archetype checklists, two-lane model, and loop-back triggers.
• TA-Data L2 (required): per-flow deep threat models are the source material for Critical-tier scenario walkthroughs.
• SA-Data L2 (required): IaC-encoded patterns and tier-conditional overlays are what L2 reviews verify the design against.
• SM-Data L2 (required): the risk-tier rubric (data classification, lineage, volume/criticality, cross-border flows, training-vs-inference use, decision-affecting use, SAR exposure) determines which flows receive scenario-based reviews and drives the per-tier drift-detection cadence.
• Supports / unblocks: IR-Data L2 (drift detected here re-opens IR), ST-Data L2 (scenario-based reviews feed the security test suite).

Desired Outcomes

• Every Critical-tier DR covers 3–5 specific threat scenarios from the TA-Data library, with the decision tied explicitly to how the proposed design handles each scenario.
• Design drift is detected quarterly for Critical-tier and annually for High-tier; material drift automatically re-opens the DR record.
• Where a Critical-tier data flow feeds a first-party AI software artifact, DR-Data and DR-Software produce joint review records with an explicit handoff boundary and shared residual-risk ownership.
• Fewer IR-stage surprises, drift caught at the design review or drift-detection phase rather than at implementation review or incident.

Activities

A) Scenario-based reviews for Critical and High-tier data flows

For every Critical-tier data flow, the full-lane checklist walkthrough is replaced by a scenario walkthrough:

• Source 3–5 specific threat scenarios from the TA-Data per-flow deep threat model and from the TA-Data archetype library. Scenarios must be specific to this flow's data classes, cross-border routing, consumer AI artifacts, and processing context, not generic archetype scenarios.
• Walk each scenario: "If an adversary does X, does the proposed design have a control that prevents or detects it? Where? What is the residual risk?" The DR decision record maps each scenario to a design control or an accepted residual risk with a named owner and expiry.
• Scenario sources include TA-Data per-flow deep threat models; anonymized IM-Data incidents from the same archetype; MITRE ATLAS data-attack techniques (AML.T0018, backdoor ML model, AML.T0019, poison training data, AML.T0025, model inversion, AML.T0037, data from information repositories); OWASP LLM entries relevant to the archetype's position in the inference chain.
• For High-tier flows: standard full-lane review augmented with at least one scenario from the TA-Data archetype library; not a full scenario walkthrough.

B) Cross-domain joint reviews for Critical-tier data flows feeding first-party AI artifacts

When a Critical-tier data flow feeds a first-party AI software artifact (e.g., a training corpus flowing to a fine-tune workload, an inference-input stream flowing to an LLM-integrated app, a retrieval store serving a RAG pipeline), DR-Data coordinates a joint review with DR-Software:

• Joint review: the DR-Data reviewer and the DR-Software reviewer attend the same session; the handoff boundary (which controls are the data-pipeline team's responsibility vs. the AI-software team's) is explicitly documented in both DR records.
• DR-Data decision covers the data-flow design; DR-Software decision covers the AI artifact's consumption design; residual risks spanning both are noted in both records with shared ownership.
• Where the first-party AI artifact is new and no DR-Software record exists, DR-Data flags the gap and holds the Sanctioned status until DR-Software completes.

C) Design-drift detection

Compare the live production data flow against its approved DR design at the published cadence:

• Critical-tier: quarterly drift check. Sources checked: data-catalog change webhooks (Atlan / Collibra / DataHub / Unity Catalog); pipeline-metadata changes (Airflow / dbt / Fivetran); lineage-graph changes; classification-label-scan deltas (Macie / BigID / Purview baseline vs. current); cross-border-flow routing changes (new country in pipeline, transfer-mechanism changes).
• High-tier: annual drift check using the same sources.
• Material drift (new data source added, classification scheme changed, cross-border routing changed, new consumer added, retention policy changed, DSAR-surface changed) automatically re-opens the DR record and routes back through the appropriate lane.
• Drift check produces a written artifact: the diff between approved design and live configuration, each delta classified as material / non-material, material deltas tracked to DR re-review or accepted residual.

Outcome Metrics (L2)

Process Metrics (leading)

• Scenario library from TA-Data refreshed quarterly; scenario content aligned to current TA-Data per-flow models.
• Drift-detection tooling health monitored, staleness alert if a Critical data flow has no drift check in the last 90 days.
• Cross-domain coordination channel with DR-Software established; joint-review calendar maintained.
• Reviewer population trained on scenario-based walkthrough technique (scenario selection, design-to-scenario mapping, residual-risk documentation).

Effectiveness Metrics (business value)

• Fewer IR-stage surprises, drift caught pre-IR, not post-deployment.
• Scenario-driven reviews produce more specific approve-with-conditions lists; conditions are more actionable than checklist items.
• Joint DR-Data / DR-Software reviews reduce handoff gaps between data governance and AI-software security decisions for Critical-tier flows.

Success Criteria

• 100% of Critical-tier DR reviews conducted as scenario-based walkthroughs with the decision tied to how the design handles each scenario.
• Design-drift detection operating for Critical (quarterly) and High (annual) data flows; 100% of material drifts re-routed to DR.
• Joint DR-Data / DR-Software review records on file for 100% of Critical-tier data flows feeding first-party AI artifacts.
• IR-stage design surprises measurably fewer than at L1 over consecutive quarters.

Maturity Level 3

Objective: Operate continuous design attestation via automated SA-Data-pattern-compliance scans, automate drift-triggered DR exception tickets, and contribute review rubrics and scenario templates to OpenSSF AI Data, DAMA, EDM Council, and CSA

At this level, Critical-tier data flows attest continuously rather than being reviewed periodically. SA-Data-pattern-compliance scans produce a daily signal that the flow's configuration and lineage metadata remain within the bounds of the approved design. Pattern drift opens a DR-exception ticket automatically. Review rubrics, scenario templates, and pattern-evolution frameworks are contributed to industry bodies. Pattern evolution is driven by external signals (MITRE ATLAS updates, GDPR enforcement actions, sector ISACs, IM-Data incidents) and internal signals (ST-Data test findings, ML-Data monitoring telemetry) on a quarterly cadence.

Dependencies

• DR-Data L2 (required): scenario reviews, drift detection, and joint-review process must be established before automation is trustworthy.
• SA-Data L3 (required): externalized patterns supply the attestation frame; automated SA-Data-pattern-compliance scans verify against the published pattern.
• ML-Data L2+ (required): monitoring signals (classification-label telemetry, lineage-graph telemetry, retention-enforcement telemetry) feed the continuous attestation pipeline.
• IM-Data L2+ (required): incidents auto-trigger DR re-examination; the IM → DR feedback loop must be operational before L3 automation is meaningful.

Desired Outcomes

• Critical-tier data flows' design posture is readable from a daily attestation signal, reviewers handle exceptions and novel flow architectures, not routine checks.
• Pattern evolution is driven quarterly by external signals (ATLAS data-attack techniques, GDPR enforcement decisions, ISACs) and internal signals (IM-Data, ST-Data, ML-Data) with a traceable change log.
• Review rubrics and scenario templates are published externally and adopted by peer organizations; the program contributes to the AI-assurance data-review ecosystem.
• DR review backlog shrinks to exception and novel-architecture work.

Activities

A) Continuous design attestation via automated SA-Data-pattern-compliance scans

• Critical-tier data flows produce a daily attestation signal covering: catalog-metadata scan (classification labels current and consistent with SM-Data taxonomy); lineage-graph check (lineage as approved, no new sources or consumers added without DR review); consent-basis currency check (GDPR Art. 6/9 lawful basis records still active and not expired); retention-enforcement check (deletion jobs running and on schedule); encryption-configuration check (keys in vault, encryption algorithms unchanged); cross-border-routing check (transfer mechanisms current, no new jurisdictions in pipeline).
• Deviations from the approved design automatically open a DR-exception ticket in IM-Data; the ticket is triaged within 3 business days.
• Attestation artifacts are machine-readable and regulator-consumable, GDPR Art. 35 DPIA evidence, EU AI Act Art. 10 data-governance records, and ISO/IEC 42001 AIMS operational records are produced by the attestation pipeline without manual assembly.
• Human reviewers handle: novel flow architectures that do not fit existing attestation rules; accepted exceptions with documented rationale; escalations from the IM-Data backlog.

B) Contribute review rubrics and scenario templates to industry

• Publish under Apache 2.0 or equivalent through OpenSSF AI Data working group, DAMA International AI Data Governance, EDM Council AI Data Risk, or CSA AI Safety Initiative:
• Per-archetype AI/HAI data design review rubric (tier-assignment criteria, checklist items with evidence pointers, scenario-selection guidance keyed to ATLAS data-attack tactics).
• Scenario template library (scenario format, per-archetype examples keyed to AML.T techniques, debrief rubric for calibration exercises).
• Pattern-evolution framework (how external signals, ATLAS updates, GDPR enforcement actions, ISACs advisories, IM-Data incidents, feed DR checklist and scenario updates on a quarterly cadence).
• Internal rubrics and templates remain aligned to the published external versions; internal deviations are proposed as upstream changes, not silently forked.
• Adoption tracked: citations, forks, direct acknowledgment from peer organizations or standards bodies.

C) Pattern evolution driven by external and internal signals

• Quarterly pattern-evolution review: external signals (MITRE ATLAS data-attack technique additions and refinements, AML.T0018 Backdoor ML Model, AML.T0019 Poison Training Data, AML.T0025 Model Inversion via ML Inference API, AML.T0037 Data from Information Repositories; GDPR enforcement decisions affecting training-data governance; sector ISAC AI-data advisories; OWASP LLM entries relevant to retrieval and inference inputs) plus internal signals (IM-Data incident patterns by archetype, ST-Data test findings, ML-Data telemetry anomalies) feed structured checklist and scenario library updates.
• Updates change-logged with signal provenance; downstream DR records for in-flight reviews notified of pattern changes that affect their archetype.
• Where a new ATLAS technique or IM-Data incident reveals a checklist gap, the gap is propagated to SA-Data and SR-Data as well, the traceability chain from threat to requirement to design review is maintained.

Outcome Metrics (L3)

Process Metrics (leading)

• Attestation-scan health, % Critical data flows producing a fresh attestation signal in the last 24 hours; staleness alert if any Critical flow silent for >48 hours.
• External-signal ingestion cadence, ATLAS and ISAC feeds processed monthly into the pattern-update queue.
• Contribution pipeline, ≥1 artifact in draft, in-review, or published at any time.
• Exception-queue freshness, DR-exception tickets triaged within 3 business days of opening.

Effectiveness Metrics (business value)

• Reviewer-hours per data flow trending down quarter-over-quarter as continuous attestation absorbs routine design-check work.
• External adoption of published rubrics and scenario templates, citations from peer organizations, sector bodies, or regulators signal industry recognition.
• Critical-incident MTTR shortened because data-flow design posture is continuously visible; incident responders do not need to reconstruct the approved design from scratch during an event.

Success Criteria

• Daily attestation operating for ≥90% of Critical-tier data flows; DR-exception tickets opened on deviation and triaged within 3 business days.
• ≥2 externally contributed review artifacts per year (per-archetype rubrics, scenario templates, pattern-evolution frameworks) with documented adoption.
• Review backlog for non-exception work inside ≤7 days; attestation has absorbed the pre-L3 routine review volume.
• Quarterly pattern-evolution cadence traceable to external (MITRE ATLAS AML.T data techniques, GDPR enforcement, ISACs) and internal (IM-Data, ST-Data, ML-Data) signals with a versioned change log.

Key Success Indicators

Level 1: - Per-archetype AI/HAI Data Design Checklists published and versioned, one per SM-Data archetype (training corpus, inference input stream, retrieval store, prompt/completion log corpus, embedding store, fine-tuning dataset, evaluation/test set), each traceable to the applicable SA-Data reference pattern, SR-Data requirements pack, and TA-Data threat snapshot; training-corpus and fine-tuning-dataset checklists include DPIA trigger assessment, poison-detection scan scheduling, and opt-out-path design; inference-input-stream checklist includes PII-redaction-edge design and no-train probe target. - Two-lane review model operational (fast-lane ≤2 BD, full-lane ≤5 BD) with named lead reviewers per archetype trained on EG-Data L1 practitioner curriculum; DPO acknowledgment required for all personal-data full-lane reviews. - ≥95% of AI/HAI data flows going to production in the last 90 days carry a completed DR decision record before pipeline build-out begins; every decision record includes the residual-risk list with named owner and expiry. - SA-Data pattern-update and SR-Data pack-update triggers wired so recurring deviations and waived requirements automatically queue SA and SR reviews; every IM-Data incident re-examines the DR record that approved the affected data flow.

Level 2: - 100% of Critical-tier DR records using scenario-based walkthroughs with 3–5 scenarios sourced from TA-Data per-flow deep models and anonymized IM-Data incidents; the DR decision is tied to how the design handles each scenario. - Design-drift detection operating quarterly for Critical and annually for High data flows, using data-catalog webhooks, pipeline-metadata changes, lineage-graph changes, classification-scan deltas, and cross-border-flow routing changes; 100% of material drifts re-routed to DR. - Joint DR-Data / DR-Software records on file for 100% of Critical-tier data flows feeding first-party AI software artifacts. - IR-stage design surprises measurably fewer over consecutive quarters.

Level 3: - ≥90% of Critical-tier data flows producing a daily automated SA-Data-pattern-compliance attestation signal; deviations auto-opening DR-exception tickets triaged within 3 business days. - ≥2 externally contributed review artifacts per year published to OpenSSF AI Data / DAMA / EDM Council / CSA with documented adoption; internal practice aligned to published versions. - Quarterly pattern-evolution loop traceable to MITRE ATLAS data-attack techniques, GDPR enforcement decisions, sector ISACs, IM-Data incidents, ST-Data findings, and ML-Data telemetry; versioned change log maintained. - Review backlog ≤7 days for non-exception items; attestation volume has replaced routine review work.

Common Pitfalls

Level 1: - ❌ Design review runs after the data pipeline has already been built, the checkpoint loses leverage because rework cost is already sunk; the review becomes a retrospective, not a gate. - ❌ Checklists are identical across archetypes, the fine-tuning-dataset checklist does not include DPIA trigger assessment, consent-tracking design, or opt-out-path because it was copy-pasted from the retrieval-store checklist. - ❌ Fast-lane becomes the default for everything, training corpora and inference-input streams processing PII slip through with a 15-minute async check rather than the full-lane architect session they require. - ❌ DPIA triggers are not assessed at DR, teams discover the DPIA requirement post-deployment during an audit or a subject-access request. - ❌ Approve-with-conditions is issued but conditions have no named owner and no expiry date, conditions sit unresolved at go-live with no enforcement path. - ❌ Cross-border transfer mechanisms are assumed rather than named, the DR record says "SCCs apply" without identifying which SCC template, which vendor, or which DPA clause. - ❌ DR decision records are not linked to the SM-Data inventory record, the program cannot answer "was this data flow reviewed?" without a manual search.

Level 2: - ❌ "Scenario-based" review is the same checklist read aloud in a meeting, same items, different format; the scenario-to-design-control mapping is never actually performed. - ❌ Scenario library is not refreshed quarterly, scenarios pulled from a 12-month-old TA-Data snapshot do not reflect the current per-flow deep model or recent IM-Data incidents. - ❌ Design-drift detection runs on a schedule but findings dead-end in a spreadsheet, no DR-exception ticket is opened; the approved design remains fiction while the live flow has diverged. - ❌ Joint DR-Data / DR-Software reviews never happen because the coordination channel with DR-Software was never established, Critical-tier training corpora flowing to fine-tune workloads have no handoff boundary documentation on file. - ❌ Per-tier drift-detection cadence exists on paper but the drift-check tooling was never wired to the data catalog, quarterly Critical-tier drift checks are reported on the scoreboard but never performed.

Level 3: - ❌ Attestation signals show green across all Critical data flows but the underlying checks cover only retention-job run status, classification-label currency, consent-basis expiry, cross-border routing, and encryption-key-vault binding are not checked; attestation is cosmetic. - ❌ Externally published rubrics diverge from internal practice, the published artifact reflects how the org reviewed data flows 18 months ago; peer adopters find inconsistencies when comparing the rubric to actual DR records. - ❌ Exception queue overwhelms reviewers because attestation thresholds are too sensitive, every minor pipeline configuration change opens a DR-exception ticket; reviewers suppress the signal source to stop the noise rather than tune the sensitivity threshold. - ❌ Industry contributions are conference talks describing the program, no technical artifacts (rubrics, scenario templates, pattern-evolution frameworks) land in OpenSSF AI Data / DAMA / EDM Council with documented adoption.

Practice Maturity Questions

Level 1: 1. Is there a published, versioned per-archetype AI/HAI Data Design Checklist, one per SM-Data archetype (training corpus, inference input stream, retrieval store, prompt/completion log corpus, embedding store, fine-tuning dataset, evaluation/test set), traceable to the applicable SA-Data reference pattern, SR-Data requirements pack, and TA-Data threat snapshot, with training-corpus and fine-tuning-dataset checklists covering DPIA trigger assessment, poison-detection scan scheduling, and opt-out-path design, and inference-input-stream checklists covering PII-redaction-edge design and no-train probe target? 2. Do ≥95% of AI/HAI data flows going to production in the last 90 days carry a completed DR decision record (approve / approve-with-conditions / send-back) before pipeline build-out begins, with a two-lane routing model (fast-lane ≤2 BD, full-lane ≤5 BD), named lead reviewers per archetype trained on EG-Data L1, DPO acknowledgment for personal-data full-lane reviews, and a residual-risk list with named owner and expiry in every record? 3. Are recurring pattern deviations and repeatedly-waived SR-Data requirements automatically queuing SA-Data pattern-update and SR-Data pack-update reviews, and does every IM-Data incident trigger a re-examination of the DR record that approved the affected data flow?

Level 2: 1. Are 100% of Critical-tier DR reviews conducted as scenario-based walkthroughs, with 3–5 specific threat scenarios sourced from TA-Data per-flow deep models and anonymized IM-Data incidents, with the DR decision tied explicitly to how the proposed design handles each scenario rather than checklist conformance alone? 2. Is design-drift detection running quarterly for Critical-tier and annually for High-tier data flows, using data-catalog change webhooks, pipeline-metadata changes, lineage-graph changes, classification-label-scan deltas, and cross-border-routing changes, with 100% of material drifts automatically re-routed to DR for a new review? 3. Are joint DR-Data / DR-Software review records on file for 100% of Critical-tier data flows feeding first-party AI software artifacts, with an explicit handoff boundary and shared residual-risk ownership documented in both DR records?

Level 3: 1. Are ≥90% of Critical-tier AI/HAI data flows producing a daily automated SA-Data-pattern-compliance attestation signal, checking classification-label currency, lineage-graph bounds, consent-basis expiry, retention-enforcement status, encryption-key-vault binding, and cross-border routing, with deviations auto-opening DR-exception tickets triaged within 3 business days? 2. Has the program contributed ≥2 substantive review artifacts per year (per-archetype rubrics, scenario templates, pattern-evolution frameworks) to OpenSSF AI Data, DAMA, EDM Council, or CSA AI Safety Initiative, with documented adoption and internal practice aligned to the published versions? 3. Is there a quarterly pattern-evolution review driven by external signals (MITRE ATLAS data-attack techniques, GDPR enforcement decisions, sector ISAC advisories) and internal signals (IM-Data incidents, ST-Data findings, ML-Data telemetry), with a versioned change log and notification to in-flight DR reviews affected by pattern changes?

Document Version: HAIAMM v3.0 Practice: Design Review (DR) Domain: Data Last Updated: 2026-05-13 Author: Verifhai