Design Review (DR) - Software Assessment

Assessment questionnaire for measuring maturity. Answer each question honestly based on current, implemented practices.

v3.0 framing: The canonical source-of-truth for Design Review (DR) in the Software domain is ../practices/DR-Software-OnePager.md. This questionnaire is authored against that one-pager. Canonical subject and through-lines: ../HAIAMM-v3.0-Framing.md §8.

Design Review (DR) - Software Domain

HAIAMM Assessment Questionnaire v3.0

Practice: Design Review (DR) Domain: Software Purpose: Assess organizational maturity in operating the design checkpoint between intake approval and build-out for every AI/HAI software artifact, confirming pattern adherence, SR coverage, and documented residual risks before engineering begins Scoring Model: Evidence + Outcome Metrics (see Scoring Methodology below)

Instructions

Answer each question honestly based on current, implemented practices (not plans or aspirations)
Each question has two components: Evidence (what you did) and Outcome Metrics (how well it worked)
Scoring uses 4 tiers: Fully Mature (1.0), Implemented (0.67), Partial (0.33), Not Implemented (0.0)
Answer progressively - Complete all Level 1 questions before Level 2
Level progression - Achieve ALL questions at lower level before advancing
Baseline first - Record current metric values before setting targets

Scoring Methodology

Score	Label	Criteria
1.0	Fully Mature	Evidence complete AND ≥3 outcome metrics meet targets
0.67	Implemented	Evidence complete AND 2 outcome metrics meet targets
0.33	Partial	Evidence partially complete OR <2 outcome metrics meet targets
0.0	Not Implemented	No evidence of practice

Level Score = average of question scores within a level Overall DR-Software Score = weighted average: L1 × 0.5 + L2 × 0.3 + L3 × 0.2

Maturity Level 1

Objective: Run a per-archetype design checkpoint for every AI/HAI software artifact before build-out, producing a written decision traceable to SA pattern, SR requirements, and TA threat snapshot

Question 1: Per-Archetype AI/HAI Software Design Checklist

Q1.1: Is there a published, versioned per-archetype AI/HAI Software Design Checklist, one per SM-Software archetype (LLM-integrated app, agent, RAG pipeline, fine-tune/training workload, eval harness, model-serving service, classical ML model), traceable to the applicable SA-Software reference pattern, SR-Software requirements pack, and TA-Software threat snapshot?

Q1.2: Does the agent checklist specifically cover tool allowlist, per-tool scope minimization, HITL gate specification (synchronous approval for destructive or external-network actions), kill-switch design, session memory bounds, and tool-call logging?

Q1.3: Are checklist items updated within 30 days of any SA-Software pattern change or SR-Software pack update?

Evidence Required: - [ ] Per-archetype checklist set published and version-controlled, one file or section per SM-Software archetype with an explicit version stamp - [ ] Agent-archetype checklist includes all six mandatory items: tool allowlist, per-tool scope minimization, HITL gate spec, kill-switch design, session memory bounds, tool-call logging - [ ] Each checklist item carries an evidence pointer and is traced to a specific SA-Software pattern control or SR-Software requirement - [ ] Common spine items present across all seven checklists: pattern adherence, data boundary, identity, logging, failure modes, disclosure, residual risk list - [ ] Change log showing checklists were updated following the most recent SA-Software or SR-Software revision - [ ] Named lead reviewer per archetype confirmed in training records

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | % AI/HAI software artifacts going to production with a completed DR decision record before build-out | % | % | ≥95% | ☐ | SM inventory × DR records | | % DR records referencing the applicable SA reference pattern and SR REM | % | % | 100% | ☐ | DR records | | Median review turnaround, fast-lane | ___ BD | ___ BD | ≤2 BD | ☐ | Review SLA telemetry | | Median review turnaround, full-lane | ___ BD | ___ BD | ≤5 BD | ☐ | Review SLA telemetry |

Metric Collection Guidance: - DR coverage: Count artifacts that reached production with a dated DR decision record before build-out start, divided by total artifacts promoted to production. Source: SM-Software inventory joined to DR record store. Measured quarterly. - Pattern and REM reference rate: Inspect DR records for a hyperlink or identifier to the SA reference pattern and SR REM. Manual spot-check or automated field validation. Measured per batch of reviews. - Fast-lane SLA: P50 of (DR decision date − review submission date) for reviews routed to fast-lane. Source: review-tracking system timestamps. - Full-lane SLA: P50 of same calculation for full-lane reviews. Track separately to detect routing imbalances.

Answer: - ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets) - ☐ Implemented (Evidence complete + 2 metrics meet targets) - ☐ Partial (Evidence partially complete + <2 metrics meet targets) - ☐ Not Implemented (No per-archetype checklist published)

Evidence Location	Validation Date	Notes

Question 2: Two-Lane Review Routing and Decision Records

Q2.1: Is a two-lane routing model operational, fast-lane (Low/Medium tier, on-pattern, ≤2 BD) and full-lane (High/Critical tier, any pattern deviation, agent archetype, or regulated data, ≤5 BD), with routing criteria published and applied consistently?

Q2.2: Does every DR decision record contain: decision (approve / approve-with-conditions / send-back); checklist completed with evidence pointers; deviations listed with rationale; residual risks with named owner and expiry; reviewer name and date; links to SM inventory record, TA threat snapshot, and SR REM?

Q2.3: Are open approve-with-conditions items tracked to resolution, with named owners, expiry dates, and an enforcement path before go-live?

Evidence Required: - [ ] Routing criteria document published specifying which tier/archetype/deviation combinations trigger full-lane vs. fast-lane - [ ] Decision record template with all required fields used for the last 10 reviews (sample auditable) - [ ] Approve-with-conditions items logged in a trackable backlog with named owner and expiry date per item - [ ] Sample of ≥5 decision records showing the residual-risk list populated (not blank) - [ ] Two-lane review SLA posted and communicated to engineering teams - [ ] Reviewer training records confirming EG-Software L1 completion for all active reviewers

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | % AI/HAI software artifacts going to production with a completed DR decision record before build-out | % | % | ≥95% | ☐ | SM inventory × DR records | | Open approve-with-conditions items aging >60 days | ___ | ___ | 0 | ☐ | Action-item backlog | | Median review turnaround, fast-lane | ___ BD | ___ BD | ≤2 BD | ☐ | Review SLA telemetry | | Median review turnaround, full-lane | ___ BD | ___ BD | ≤5 BD | ☐ | Review SLA telemetry |

Metric Collection Guidance: - Approve-with-conditions aging: Query the action-item backlog for items where (today − condition creation date) > 60 days and status is not resolved. Source: JIRA / Linear / equivalent. Measured weekly. - DR coverage: same as Q1 guidance above. - SLA telemetry: same as Q1 guidance above. Alert if P90 fast-lane exceeds 3 BD or full-lane exceeds 7 BD.

Evidence Location	Validation Date	Notes

Question 3: Loop-back to SA-Software, SR-Software, and IM-Software

Q3.1: Are recurring pattern deviations and repeatedly-waived SR requirements automatically queuing SA pattern-update and SR pack-update reviews, with a threshold of three deviations in the same direction for the same archetype triggering a pattern-update review?

Q3.2: Does every IM-Software incident trigger a re-examination of the DR decision record that approved the affected artifact, specifically asking which checklist item would have caught the issue?

Q3.3: Is the pattern-deviation rate tracked by archetype and surfaced in a regular program review?

Evidence Required: - [ ] Documented trigger rule: three same-direction deviations per archetype auto-queues SA pattern-update review - [ ] SA pattern-update queue showing items queued from DR feedback in the last 12 months - [ ] SR pack-update queue showing items queued from repeatedly-waived requirements in the last 12 months - [ ] IM-Software incident post-mortems that include a section linking back to the DR decision record - [ ] Checklist updated in response to at least one IM-Software incident finding (with before/after version comparison) - [ ] Pattern-deviation rate dashboard or report by archetype reviewed in the last quarter

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | % DR decision records referencing the applicable SA reference pattern and SR REM | % | % | 100% | ☐ | DR records | | SA/SR update items queued from DR feedback in last 12 months | ___ | ___ | ≥1 | ☐ | SA/SR update queues | | % IM-Software incidents with a DR record re-examination step | % | % | 100% | ☐ | IM post-mortems | | Open approve-with-conditions items aging >60 days | ___ | ___ | 0 | ☐ | Action-item backlog |

Metric Collection Guidance: - SA/SR queue items from DR: Count items in the SA pattern-update queue or SR pack-update queue whose "source" field references DR feedback. Measured quarterly. - IM incident DR re-examination: Review IM-Software incident records and confirm each has a DR-record field populated. Track coverage as a percentage. Measured after each incident closes.

Evidence Location	Validation Date	Notes

Maturity Level 2

Objective: Upgrade Critical-tier reviews to scenario-based walkthroughs driven by TA-Software per-artifact models, detect design drift for High and Critical artifacts on a published cadence, and run joint DR-Software / DR-Vendors reviews for Critical-tier artifacts integrating with vendor AI

Question 4: Scenario-Based Reviews for Critical and High-Tier Artifacts

Q4.1: Are 100% of Critical-tier DR reviews conducted as scenario-based walkthroughs, with 3–5 specific threat scenarios sourced from the TA-Software per-artifact deep threat model and anonymized IM-Software incidents, with the DR decision tied explicitly to how the proposed design handles each scenario?

Q4.2: Are scenario sources refreshed quarterly from the TA-Software per-artifact deep models, MITRE ATLAS techniques (TA0006/TA0007/TA0008 for agent archetype primary coverage; TA0001/TA0003 where design affects initial-access surface), and OWASP LLM / Agentic Top 10?

Q4.3: For High-tier artifacts, is the standard full-lane review augmented with at least one scenario from the TA archetype library?

Evidence Required: - [ ] DR records for Critical-tier artifacts showing scenario-based walkthrough format with ≥3 named scenarios per review - [ ] Each scenario in the record maps to a design control or an accepted residual risk with named owner and expiry - [ ] Scenario library version-controlled with quarterly refresh dates and signal provenance - [ ] TA-Software per-artifact deep threat model referenced in each Critical-tier DR record - [ ] High-tier DR records showing at least one augmenting scenario from the TA archetype library - [ ] Reviewer population trained on scenario-based walkthrough technique

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | % Critical-tier DR records using scenario-based walkthrough | % | % | 100% | ☐ | DR records | | % Critical/High-tier artifacts with drift check on published cadence | % | % | ≥95% | ☐ | Drift-check schedule × SM inventory | | % material drift findings re-routed to DR | % | % | 100% | ☐ | Drift-detection queue | | IR-stage design surprises (findings at IR with no corresponding DR condition) | ___ | ___ | trending down | ☐ | IR records |

Metric Collection Guidance: - Scenario-based coverage: Count Critical-tier DR records with a "scenarios" section listing ≥3 named threat scenarios, divided by total Critical-tier DR records in the period. Source: DR record store. Measured quarterly. - IR-stage surprises: Count IR findings that have no corresponding approve-with-conditions or residual-risk entry in the DR record for the same artifact. Source: IR records joined to DR records. Trend tracked quarter-over-quarter.

Evidence Location	Validation Date	Notes

Question 5: Design-Drift Detection for High and Critical Artifacts

Q5.1: Is design-drift detection operating quarterly for Critical-tier and annually for High-tier artifacts, using code-repository changes, model-registry events, deploy-event configs, CI/CD parameters, and IaC state, with findings documented as material or non-material?

Q5.2: Are material drift findings (new tool added to an agent, new data class flowing into RAG or fine-tune, model family changed, new retrieval source, new region) automatically re-opening the DR record and routing back through the appropriate lane?

Q5.3: Does the drift-detection tooling produce a staleness alert if a Critical artifact has no drift check in the last 90 days?

Evidence Required: - [ ] Drift-detection schedule showing Critical artifacts checked quarterly and High artifacts annually - [ ] Drift check artifacts (written diffs) for ≥3 Critical-tier artifacts in the last 12 months - [ ] Classification criteria document defining which delta types are material vs. non-material - [ ] At least one material drift finding that re-opened a DR record and routed to a new review - [ ] Staleness alert configuration confirmed (Critical artifact silent for >90 days triggers alert) - [ ] Drift-detection sources confirmed: code repo, model registry, deploy events, CI/CD, IaC

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | % Critical/High-tier artifacts with drift check on published cadence | % | % | ≥95% | ☐ | Drift-check schedule × SM inventory | | % material drift findings re-routed to DR | % | % | 100% | ☐ | Drift-detection queue | | % Critical-tier DR records using scenario-based walkthrough | % | % | 100% | ☐ | DR records | | IR-stage design surprises (findings at IR with no corresponding DR condition) | ___ | ___ | trending down | ☐ | IR records |

Metric Collection Guidance: - Drift check cadence: Count Critical artifacts with a documented drift check in the last 90 days, divided by total Critical artifacts in the SM inventory. Measured monthly. - Material drift re-routing: Count material drift findings that have a corresponding DR re-review record, divided by total material drift findings. Measured per drift-check cycle.

Evidence Location	Validation Date	Notes

Question 6: Joint DR-Software / DR-Vendors Reviews for Critical-Tier Vendor AI Integrations

Q6.1: Are joint DR-Software / DR-Vendors review records on file for 100% of Critical-tier first-party artifacts integrating with vendor AI services, with an explicit handoff boundary documented in both DR records?

Q6.2: Do both DR records share residual-risk ownership, with risks spanning both the first-party artifact and the vendor integration named in both records, assigned to a single named resolution owner?

Q6.3: Where a vendor integration is new and no DR-Vendors record exists, does DR-Software hold the artifact's Sanctioned status until DR-Vendors completes?

Evidence Required: - [ ] Joint review calendar or coordination log showing DR-Software and DR-Vendors reviewers attending the same session for Critical-tier vendor integrations - [ ] DR-Software decision records for Critical-tier vendor integrations explicitly referencing the corresponding DR-Vendors record identifier - [ ] Handoff boundary documented in both records (first-party responsibilities vs. vendor responsibilities) - [ ] Shared residual risks noted in both records with a single named resolution owner - [ ] Sanctioned-status hold mechanism confirmed, no Critical-tier artifact with a new vendor integration advanced without a DR-Vendors record on file - [ ] Cross-domain coordination channel with DR-Vendors established (Slack channel, shared calendar, or equivalent)

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | % Critical-tier artifacts integrating with vendor AI with a joint DR-Software / DR-Vendors record | % | % | 100% | ☐ | DR records × vendor integration tracker | | % material drift findings re-routed to DR | % | % | 100% | ☐ | Drift-detection queue | | % Critical-tier DR records using scenario-based walkthrough | % | % | 100% | ☐ | DR records | | IR-stage design surprises (findings at IR with no corresponding DR condition) | ___ | ___ | trending down | ☐ | IR records |

Metric Collection Guidance: - Joint record coverage: Count Critical-tier software artifacts with a vendor AI integration that have a paired DR-Vendors record identifier in their DR-Software record, divided by total such artifacts. Source: DR record store joined to vendor integration tracker. Measured quarterly. - Shared residual risk completeness: Spot-check 5 joint reviews per quarter, verify shared residual risks appear in both records with a single named owner.

Evidence Location	Validation Date	Notes

Maturity Level 3

Objective: Operate continuous design attestation via automated SA-pattern-compliance scans, automate drift-triggered DR exception tickets, and contribute review rubrics and scenario templates to OpenSSF AI, CSA, and OWASP SAMM AI

Question 7: Continuous Design Attestation via SA-Pattern-Compliance Scans

Q7.1: Are ≥90% of Critical-tier AI/HAI software artifacts producing a daily automated SA-pattern-compliance attestation signal, checking code-repo control presence, model-registry bounds, IaC drift, and logging completeness, with deviations automatically opening DR-exception tickets triaged within 3 business days?

Q7.2: Are attestation artifacts machine-readable and regulator-consumable, producing EU AI Act Art. 9 risk-management evidence and ISO/IEC 42001 AIMS operational records without manual assembly?

Q7.3: Do human reviewers handle only: novel architectures not covered by existing attestation rules, accepted exceptions with documented rationale, and IM-Software escalations?

Evidence Required: - [ ] Attestation pipeline configuration showing daily scan cadence for Critical-tier artifacts - [ ] Coverage report: % of Critical-tier artifacts producing a fresh attestation signal in the last 24 hours - [ ] Sample attestation artifact (machine-readable format) showing all four check domains: code-repo, model-registry, IaC, logging completeness - [ ] DR-exception ticket queue showing tickets opened automatically on attestation deviation - [ ] Evidence that at least one DR-exception ticket was triaged within 3 business days of opening - [ ] Staleness alert configuration confirmed (Critical artifact silent for >48 hours triggers alert)

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | % Critical-tier artifacts producing a daily attestation signal | % | % | ≥90% | ☐ | Attestation telemetry | | Mean DR-exception ticket age from open to triage | ___ BD | ___ BD | ≤3 BD | ☐ | DR-exception queue | | Review backlog age, non-exception items | ___ days | ___ days | ≤7 days | ☐ | Review queue telemetry | | Quarterly pattern-evolution reviews conducted | ___ | ___ | 4 / year | ☐ | Pattern-update log |

Metric Collection Guidance: - Attestation coverage: Count Critical artifacts with a completed attestation scan in the last 24 hours, divided by total Critical artifacts. Sourced from attestation pipeline run log. Measured daily; alert if below 90%. - Exception ticket SLA: P50 of (triage timestamp − open timestamp) for DR-exception tickets. Source: JIRA / Linear. Measured weekly. - Review backlog age: P90 of age of non-exception review queue items. Source: review-tracking system. Measured weekly.

Evidence Location	Validation Date	Notes

Question 8: Contribute Review Rubrics and Scenario Templates to Industry

Q8.1: Has the program contributed ≥2 substantive review artifacts per year (per-archetype rubrics, scenario templates, pattern-evolution frameworks) to OpenSSF AI, CSA AI Safety Initiative, or OWASP SAMM AI, with documented adoption by peer organizations or standards bodies?

Q8.2: Are internal rubrics and templates kept aligned to the published external versions, with internal deviations proposed as upstream changes rather than silently forked?

Q8.3: Is adoption tracked via citations, forks, or direct acknowledgment from peer organizations or standards bodies?

Evidence Required: - [ ] Contribution log showing ≥2 published artifacts (rubrics, scenario templates, or pattern-evolution frameworks) in the last 12 months - [ ] Publication links to OpenSSF AI, CSA AI Safety Initiative, OWASP SAMM AI, or equivalent, Apache 2.0 or equivalent license - [ ] Adoption evidence: citations, GitHub forks, or written acknowledgment from a peer organization or standards body - [ ] Internal rubric version compared to external published version, confirmed aligned or upstream PR submitted for divergences - [ ] At least one artifact in draft, in-review, or published at the time of assessment

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | Industry contributions per year (rubrics, scenario templates, pattern-evolution frameworks) | 0 | ___ | ≥2 | ☐ | Contribution log | | % Critical-tier artifacts producing a daily attestation signal | % | % | ≥90% | ☐ | Attestation telemetry | | Mean DR-exception ticket age from open to triage | ___ BD | ___ BD | ≤3 BD | ☐ | DR-exception queue | | Quarterly pattern-evolution reviews conducted | ___ | ___ | 4 / year | ☐ | Pattern-update log |

Metric Collection Guidance: - Industry contributions: Count distinct published artifacts (not versions of the same artifact) that are publicly accessible under an open license and attributed to this organization. Measured annually. - Adoption evidence: Log citations and forks quarterly. A citation is a reference by a named organization in a public document; a fork is a documented derivative under the upstream license.

Evidence Location	Validation Date	Notes

Question 9: Quarterly Pattern Evolution Driven by External and Internal Signals

Q9.1: Is there a quarterly pattern-evolution review driven by external signals (MITRE ATLAS updates for TA0006/TA0007/TA0008 and TA0001; sector ISAC advisories; OWASP LLM / Agentic Top 10 revisions) and internal signals (IM-Software incident patterns, ML-Software telemetry anomalies, ST-Software red-team findings), with a versioned change log?

Q9.2: Are downstream DR records for in-flight reviews notified of pattern changes that affect their archetype?

Q9.3: Where a new ATLAS technique or IM-Software incident reveals a checklist gap, is the gap propagated to SA-Software and SR-Software to maintain the traceability chain?

Evidence Required: - [ ] Quarterly pattern-evolution review calendar with at least 4 sessions completed in the last 12 months - [ ] Change log for the checklist and scenario library showing signal provenance (which external or internal signal triggered each update) - [ ] Evidence that ATLAS technique additions for TA0006/TA0007/TA0008 and TA0001 were reviewed in the last quarter - [ ] In-flight DR review notifications sent when a pattern change affected the archetype under review - [ ] SA-Software and SR-Software update items queued from pattern-evolution checklist gaps - [ ] ISAC and ATLAS feeds ingested on a monthly cadence into the pattern-update queue

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | Quarterly pattern-evolution reviews conducted | ___ | ___ | 4 / year | ☐ | Pattern-update log | | % Critical-tier artifacts producing a daily attestation signal | % | % | ≥90% | ☐ | Attestation telemetry | | Industry contributions per year | 0 | ___ | ≥2 | ☐ | Contribution log | | Review backlog age, non-exception items | ___ days | ___ days | ≤7 days | ☐ | Review queue telemetry |

Metric Collection Guidance: - Pattern-evolution cadence: Count completed quarterly reviews with a dated agenda, attendees, and change log entry. Source: meeting records and change log. Measured annually. - Signal provenance completeness: Spot-check 5 change log entries per quarter, verify each has a named signal source (ATLAS technique ID, ISAC advisory reference, or IM incident ID).

Evidence Location	Validation Date	Notes

Summary Scorecard

Question	Level	Score (0 / 0.33 / 0.67 / 1.0)	Notes
Q1: Per-Archetype Checklist	L1
Q2: Two-Lane Routing and Decision Records	L1
Q3: Loop-back to SA / SR / IM	L1
Q4: Scenario-Based Reviews (Critical/High)	L2
Q5: Design-Drift Detection	L2
Q6: Joint DR-Software / DR-Vendors Reviews	L2
Q7: Continuous Design Attestation	L3
Q8: Industry Contributions	L3
Q9: Quarterly Pattern Evolution	L3

Level 1 Score: ___ / 1.0 (average of Q1–Q3) Level 2 Score: ___ / 1.0 (average of Q4–Q6) Level 3 Score: ___ / 1.0 (average of Q7–Q9) Overall DR-Software Score: ___ / 1.0 (L1 × 0.5 + L2 × 0.3 + L3 × 0.2)

Current Maturity Level: ☐ L1 ☐ L2 ☐ L3 Assessment Date: Assessor: Next Review Date:

Document Version: HAIAMM v3.0 Practice: Design Review (DR) Domain: Software Questionnaire Date: 2026-05-15 Author: Verifhai

Instructions:

Answer based on current practices, not plans
“Yes” requires documented evidence
Complete all Level 1 questions before Level 2
Partial implementation = “No”

↓ Download as Markdown