Implementation Review (IR)

Software Domain - HAIAMM v3.0

Practice Overview

Objective: Verify, at go-live and on a recurring cadence, that the actual code and configuration of AI/HAI software the organization builds matches the design approved at DR, and that it stays there as the artifact evolves.

Description: IR-Software is the configuration and code check for first-party AI/HAI software artifacts, the moment a reviewer opens the codebase, the deployed configuration, the model registry, and the CI/CD pipeline and confirms that what is running matches the DR decision record. At L1 the review runs at go-live, at least annually, and on material change (model swap, new tool added to an agent, new data class flowing into RAG or fine-tune, customer-exposure switched on). At L2, IR consumes code-repo webhooks, model-registry events, deploy-event telemetry, IaC scan tooling, and vendor admin APIs to detect configuration drift continuously for High and Critical-tier artifacts. Findings are severity-tagged and SLA-bound per the SM L2 tier-treatment matrix; they feed IM-Software for tracking and resolution. No-train assertions, retention settings, tool allowlists, and kill-switch functionality are probed recurrently, not trusted from design text alone.

Context: The gap between the approved design and the running system is the primary source of silent security exposure in AI/HAI software. An agent's tool allowlist is correct in the DR record but widened at deploy time by a CI/CD environment variable. A fine-tune workload's no-train assertion appears in the SR REM but the vendor admin-console setting was reset by a product update. A kill-switch is documented in the SA pattern but the implementation test was skipped during the go-live crunch. IR-Software closes these gaps by making the implementation check systematic, evidence-based, and recurring, not a one-time pre-launch checkbox or a scramble when an incident reveals a configuration regression.

Maturity Level 1

Objective: Run per-archetype implementation reviews at go-live, annually, and on material change, verifying code matches the SA pattern, config matches the DR decision, and the SR REM evidence is current

At this level, the gap between approved design and running system is systematically checked at the moments it matters most. Every review produces findings with severity tags, named owners, and SLA-bound resolution dates.

Dependencies

• DR-Software L1 (required): the approved DR decision record is the specification IR checks against; without it there is no authoritative baseline.
• SR-Software L1 (required): the REM defines which requirements must be evidenced; IR verifies the evidence is current and accurate.
• SA-Software L1 (required): the SA reference pattern defines the intended configuration shape; IR checks adherence to the pattern's controls.
• EG-Software L1 (required): reviewers must understand AI/HAI software archetypes and know where to look for pattern controls in code, config, and admin consoles.
• Supports / unblocks: ST-Software L1 (tests run against the verified configuration), EH-Software L1 (hardening acts on IR findings), IM-Software L1 (IR findings become issues in the backlog), ML-Software L1 (logging configuration verified here feeds monitoring).

Desired Outcomes

• The gap between the DR-approved design and the live configuration stays small and short-lived.
• Material configuration and code drift is found by the program, not by an external auditor or an incident.
• Every finding has a named owner, a severity tag, and a SLA-bound resolution date; aging findings are visible to the program sponsor.
• Material changes to a production AI/HAI software artifact always trigger a review before the change is in production, model swaps, new tools, new data classes, and scope changes do not bypass the gate.
• The SR REM is kept current, evidence cited in the REM is verified to still be accurate, not just present.

Activities

A) Publish the per-archetype implementation review checklist

One checklist per SM-Software archetype, focused on the configuration and code points where production reality most commonly drifts from the approved design. Each item is a yes/no with a required evidence artifact (screenshot, config export, code-review finding, test result, log sample).

Common spine across all archetypes: - Code matches SA pattern, the SA reference pattern's structural controls are present and unmodified in the deployed codebase (output filter wired and active; secrets routed through vault, not hardcoded; SSO enforcement present; egress allowlist applied). - Config matches DR decision, deployed environment variables, model-provider settings, feature flags, and IaC state match the DR decision record; deviations flagged. - SR REM evidence is current, a stratified sample of REM rows verified against current observable reality: logging confirmed via log-export test, no-train setting confirmed in vendor admin console (not from DPA text alone), secrets-scan result current with zero findings, kill-switch confirmed to function (test record on file with date). - Logging is actually producing the events the design promised, pull a sample of prompt/completion/tool-call/admin-audit events from the logging pipeline; confirm they are present and match the format and retention policy specified in the SR REM. - Kill-switches and circuit breakers actually work when triggered, execute the kill-switch test and record the result; confirm the agent process stops or the inference endpoint becomes unreachable within the specified SLA.

Archetype-specific additions:

• LLM-integrated app: output filter is active (test with a known-bad completion and confirm it is blocked or flagged); per-customer context isolation is enforced (confirm no cross-tenant prompt leakage in a multi-tenant deployment); API keys are vault-backed (confirm via secrets scan in CI and in the deployed container image).

• AI agent: tool allowlist is actually enforced in code (not only declared in the DR checklist, verify the allowlist-enforcement logic in the deployed codebase, not just the config); per-tool argument schema validation is present and applied; HITL gate is wired and fires for the declared trigger conditions (test with an out-of-scope action and confirm the gate blocks execution); kill-switch stops all tool invocation within the specified SLA; session memory bounds are enforced (confirm context does not persist across session boundaries); tool-call logging is capturing full args and return values (pull a sample from the logging pipeline). ATLAS drift sources checked: TA0006 Persistence (session memory bounds still enforced), TA0007 Privilege Escalation (tool allowlist and per-tool scope not widened since DR), TA0008 Defense Evasion (tool-call logging and kill-switch not disabled).

• RAG pipeline: retrieval-source allow-list matches the DR-approved corpus sources (no new sources added without change review); per-tenant retrieval isolation enforced at query time (test with a cross-tenant query and confirm rejection); injection-defense structure present in the deployed prompt template (retrieved content in a designated untrusted block, not concatenated with instructions).

• Fine-tune / training workload: no-train assertion confirmed via vendor admin-console API state (OpenAI / Anthropic / Bedrock / Vertex admin APIs), not from DPA text alone; training-data provenance records match the artifact's SR REM for all datasets used in the last training run; training-job isolation confirmed (training environment cannot reach production network during a training run); eval gating confirmed (model registry shows the deployed model has a current eval attestation from the eval harness).

• Eval harness: data isolation confirmed (eval logs not co-mingled with training pipelines); access control confirmed (execution rights restricted to named personnel, verify against current IAM or role assignments); findings disposition confirmed (last 30 days of eval findings are present in the IM-Software backlog with severity tags).

• Model-serving service: deployed model version matches the SM inventory record and the DR-approved version pin; rollback playbook tested within the last quarter (test record on file); canary deployment configuration matches the DR-approved rollout plan; rate-limiting and abuse-detection active (confirm via a rate-limit probe).

• Classical ML model: model provenance (training data version, feature set, model version) matches the model-registry record and the SM inventory entry; inference-endpoint access control confirmed (no unauthenticated inference via a test probe); per-request logging active (pull a sample); drift-monitoring baseline confirmed as current.

B) Perform reviews at the right moments

Three triggers at L1:

• Go-live review: before the artifact enters production (or before a new version goes live), verify the as-built code and configuration against the DR-approved design. No production cutover with a blocker finding open.
• Annual review: every active AI/HAI software artifact reviewed at least annually; scheduled from the SM inventory (last-IR-date field linked to a review-due alert).
• Material-change review: any of the following triggers an ad-hoc review before the change ships to production: model version or model family swap; new tool added to an agent's allowlist; new data class flowing into RAG corpus, fine-tune dataset, or inference input; customer-exposure switched on for a previously internal feature; new retrieval source added; new region or data-residency zone; agent scope or permission model changed. The material-change trigger is wired to the same signal sources as SM inventory material-change events.

Reviews are evidence-based, screenshots, config exports, or code-review findings stored with the IR record. Target timebox: 30–90 minutes per artifact depending on archetype complexity.

C) Track findings to closure

Every review produces zero or more findings. Each finding carries: - Severity: Critical (e.g., agent acting on customer accounts with broader tool scope than DR-approved) / High / Medium / Low. Severity is calibrated to the SM L2 tier-treatment matrix's IM SLA column; at L1 use a consistent judgment rubric pending SM L2 formalization. - Owner: named engineer or team owner; not "the engineering team." - SLA from SM tier-treatment matrix: Critical blocker resolved before production cutover or rollback required; High ≤7 days; Medium ≤30 days; Low ≤90 days or accepted residual. - Evidence: after-fix evidence artifact (screenshot, scan result, code-review comment, test result) linked to the finding before closure.

Findings feed IM-Software as issues (for tracking, aging, trending across the portfolio) and loop back to SR-Software where a finding reveals that an REM row's cited evidence was inaccurate, the REM row is updated before the finding is closed.

Drift sources verified at L1 (without continuous tooling): - Code repository: review commits since the last IR that touch SA-pattern-control files (output filter, tool allowlist, session bounds, vault bindings, secrets, egress config). - Model registry: model version and fine-tune lineage changes since the last IR. - Deploy events: configuration changes at deploy time (environment variables, feature flags, IaC outputs) since the last IR. - CI/CD job parameters: build-job parameters affecting model versions, provider endpoints, or security controls. - IaC state: Terraform / Pulumi planned vs. applied state for the artifact's infrastructure.

Outcome Metrics (L1)

Process Metrics (leading)

• Annual review calendar populated from the SM inventory; artifacts nearing review-due date visible in advance.
• Material-change trigger wired to SM inventory material-change events; reviews queued within 5 business days of a confirmed material change.
• Reviewer backlog aging, no single reviewer more than 3 artifacts overdue.
• SR REM update loop active, % of IR findings that trigger an REM row update for the affected requirement (expected to rise as the IR / SR feedback loop matures).

Effectiveness Metrics (business value)

• Drift-caught-early rate, findings closed before they reach an incident or an external audit.
• REM accuracy improvement, % of REM rows where the cited evidence was verified current vs. stale; stale-evidence rate trends down as the IR / SR feedback loop operates.
• Avoided-incident stories, documented cases where IR caught a code or configuration regression before it was exploited or caused a data event.

Success Criteria

• Per-archetype IR checklists published, owned, and linked from the SM inventory record and the DR decision record.
• Go-live, annual, and material-change review triggers wired to the SM inventory; 100% of new AI/HAI software artifacts in the last 90 days have a go-live IR record.
• ≥90% of active AI/HAI software artifacts carry a current-year IR record.
• All Critical / blocker findings resolved before production cutover; High findings closed within 7 days with evidence linked.
• Findings-aging dashboard reviewed at least monthly by the program sponsor.

Maturity Level 2

Objective: Detect configuration drift continuously for Critical and High-tier artifacts via code-repo webhooks, model-registry events, IaC scan tooling, and vendor admin APIs; probe no-train and retention settings recurrently; calibrate IR cadence per SM tier

At this level, implementation review stops being a point-in-time check and becomes a continuous signal for Critical and High-tier artifacts. Drift sources are wired to automated detection. No-train and retention settings are validated recurrently via vendor admin APIs, not trusted from DPA text. Tool scope actually exercised in a boundary test, not assumed from the allowlist declaration. IR cadence is explicitly differentiated by SM L2 risk tier.

Dependencies

• IR-Software L1 (required): per-archetype checklists and findings workflow must be established.
• SM-Software L2 (required): the risk-tier rubric drives IR cadence and depth per the tier-treatment matrix (Critical: go-live + semi-annual + continuous drift; High: go-live + annual + material change; Medium: go-live + annual; Low: go-live).
• SA-Software L2 (required): IaC-encoded patterns establish the "correct" baseline that continuous drift detection measures against.
• Supports / unblocks: ST-Software L2 (tests run against the continuously verified configuration), EH-Software L2, ML-Software L2 (monitoring configuration verified here feeds detections).

Desired Outcomes

• Configuration drift on Critical-tier artifacts is detected within days, not months.
• No-train and retention settings are verified via vendor admin API state (OpenAI / Anthropic / Bedrock / Vertex admin APIs), not trusted from DPA text alone, and are re-verified recurrently, not only at go-live.
• Tool scope for agent artifacts is tested by exercising the tool with an out-of-scope argument and verifying rejection, not assumed from the allowlist declaration.
• IR cadence visibly differentiates by tier: Critical gets semi-annual reviews plus continuous drift detection; Low gets go-live only.
• All Critical-tier IR findings carry severity tags and SLAs matching the SM L2 tier-treatment matrix.

Activities

A) Continuous drift detection from code-repo, model-registry, IaC, and deploy telemetry

Wire the following signal sources to an automated drift-detection pipeline for Critical and High-tier artifacts:

• Code-repo webhooks: commits to files touching SA-pattern-control logic (output filter, tool allowlist, session bounds, vault bindings, HITL gate, kill-switch, egress config, prompt template) trigger an automated diff against the DR-approved baseline; material deviations open an IR finding automatically.
• Model-registry events: model version promotions, fine-tune lineage changes, and model deprecation events trigger an IR re-review gate for the affected artifact; a model swap without a corresponding DR material-change review is a Critical finding.
• IaC scan tooling: Terraform / Pulumi / CloudFormation plan-vs-apply drift scans run on each deploy for Critical and High artifacts; configuration deviations from the approved IaC module (from SA-Software L2) open IR findings.
• Deploy-event telemetry: environment variable changes, feature flag changes, and secrets-rotation events captured at deploy time; changes to SA-pattern-control variables since the last IR open findings automatically.
• CI/CD job parameter telemetry: build-job parameters (model version pins, provider endpoint references, security-control toggles) compared against the DR-approved specification; deviations flagged.

Detection latency targets: Critical-tier drift detection ≤7 days from change event to finding opened; High-tier ≤30 days.

B) Vendor admin API probing for no-train and retention settings

No-train and retention settings are probed recurrently via vendor admin APIs for Critical and High-tier artifacts, not trusted from DPA text or one-time admin-console screenshots:

• OpenAI: Org Settings API, confirm data_controls.training_data_sharing is false for all applicable API keys; confirm retention period matches DPA.
• Anthropic: Organization admin settings API, confirm model training usage terms reflect no-train commitment.
• Amazon Bedrock: AWS Service Control Policy and Bedrock model invocation logging config, confirm no model fine-tuning on customer data paths; confirm CloudTrail logging is active for all Bedrock API calls.
• Google Vertex AI / Gemini: Google Cloud Organization Policy constraints, confirm aiplatform.disableModelEvaluation and data logging settings; confirm no training-data usage opt-in is active.
• Other vendors: equivalent admin API or authenticated API endpoint where available; UI-based verification with screenshot evidence as fallback where APIs are not available.

Probing cadence: Critical-tier, monthly; High-tier, quarterly. Delta from the previous probe opens an IR finding with severity matching the data-class impact of the change.

C) Tool-scope boundary testing for agent artifacts

For every Critical and High-tier agent artifact, tool-scope boundaries are tested at each IR cycle, not assumed from the tool allowlist declaration:

• Call each declared tool with an out-of-scope argument (a file path outside the declared directory scope, a record ID outside the declared customer scope, an action flagged as destructive without a HITL gate) and verify rejection. Document the test input, expected behavior, and actual behavior.
• Attempt to invoke a tool not on the allowlist from within the agent session; verify the call is blocked at the allowlist-enforcement layer, not only rejected downstream.
• Trigger the HITL gate condition for a declared destructive action; verify the gate fires synchronously and that the timeout-and-fallback behavior matches the DR specification.
• Trigger the kill-switch; verify the agent process and all active tool invocations stop within the specified SLA; record the test date and result.

Tool-scope test evidence stored with the IR record; failures are Critical findings for Critical-tier agents and High findings for High-tier agents.

D) Tier-calibrated IR cadence

Publish and enforce per the SM L2 tier-treatment matrix: - Critical: go-live + semi-annual + material-change-triggered + continuous drift detection. - High: go-live + annual + material-change-triggered. - Medium: go-live + annual. - Low: go-live + re-review on material change.

Every artifact in the SM inventory has a last-IR-date and next-IR-due field; Critical-tier artifacts with no IR in the last 180 days are escalated to the program sponsor.

Outcome Metrics (L2)

Process Metrics (leading)

• Drift-detection pipeline health monitored, % Critical artifacts producing a fresh signal in the last 7 days; on-call alert if feed silent for >48 hours.
• Vendor API probing calendar maintained; missed probes tracked as process-metric failures.
• Tool-scope test coverage tracked per agent artifact per IR cycle.
• IR backlog tier-aware; Critical-tier findings never wait behind Low-tier queue items.

Effectiveness Metrics (business value)

• Drift caught before ST/ML detections or incidents, trend measured over quarters.
• Reduced audit findings on configuration claims, external auditors (SOC 2, ISO 42001) find IR evidence sufficient without supplemental screenshots or interviews.
• No-train probing reveals vendor-side changes (admin-console setting reset by product update, retention period lengthened by plan change) before they become data-handling incidents.

Success Criteria

• ≥90% of Critical-tier artifacts under continuous drift detection; median detection latency ≤7 days.
• No-train and retention settings verified via vendor admin APIs for ≥80% of Critical/High-tier artifacts on a monthly (Critical) and quarterly (High) probing cadence.
• 100% of Critical/High-tier agent artifacts with tool-scope boundary tests and kill-switch test on record in the current IR cycle.
• Tier-cadence adherence ≥95%; Critical-tier findings aged per the SM L2 tier-treatment matrix SLAs.

Maturity Level 3

Objective: Continuous configuration attestation for Critical-tier artifacts, daily attestation signal confirming pattern compliance and evidence freshness, automatic IM ticket on drift, and contribution to OpenSSF AI reference attestation schemas and OWASP SAMM AI

At this level, configuration for Critical-tier AI/HAI software artifacts is not periodically reviewed, it is attested continuously. Every Critical artifact produces a daily attestation signal confirming that pattern compliance, REM evidence freshness, and deployed configuration are within tolerance. Drift opens an IM-Software ticket automatically. Per-archetype configuration baseline schemas are contributed to OpenSSF AI, OWASP SAMM AI, and CSA AI Safety Initiative.

Dependencies

• IR-Software L2 (required): API-based monitoring, vendor admin API probing, tool-scope testing, and tier-calibrated cadence must be in place.
• SA-Software L3 (required): externalized patterns supply the attestation frame for automated compliance scans.
• ML-Software L2+ (required): runtime signals (logging completeness, anomaly detections) are evidence sources the attestation pipeline reads.
• SR-Software L3 (alignment): machine-readable REM schema at SR L3 provides the evidence-freshness signals the attestation pipeline validates against.

Desired Outcomes

• Every Critical-tier AI/HAI software artifact produces a daily attestation signal, pattern compliance, evidence freshness, and deployed configuration are continuously within tolerance.
• Drift automatically opens an IM-Software ticket; the program does not wait for the next scheduled review to act.
• Per-archetype configuration baseline schemas (what "correct" looks like for each archetype at each tier) are published to OpenSSF AI, OWASP SAMM AI, and CSA AI Safety Initiative, the program is a net contributor to the AI-assurance implementation-review ecosystem.
• IR reviewer-hours per artifact trend down as attestation absorbs routine checks; reviewers focus on novel configurations and exception escalations.

Activities

A) Daily attestation signal for Critical-tier artifacts

Each Critical-tier AI/HAI software artifact produces a daily composite attestation signal covering three dimensions:

1. Pattern compliance: automated SA-pattern-compliance scan confirms key controls are present and active in the deployed codebase (output filter status, tool allowlist enforcement presence, vault binding status, HITL gate logic, egress allowlist configuration, session memory bound enforcement). Uses the code-repo and IaC scan tooling from IR L2, now running on a daily schedule with a machine-readable output.

2. Evidence freshness: the SR REM's evidence citations are checked for staleness, are the cited evidence artifacts (secrets-scan result, kill-switch test record, no-train API probe result, tool-scope boundary test record) within their defined freshness window? Stale evidence opens a finding automatically. Evidence freshness windows (per SM tier-treatment matrix): secrets scan ≤7 days; kill-switch test ≤90 days; no-train API probe ≤30 days (Critical) / ≤90 days (High); tool-scope boundary test ≤180 days.

3. Configuration within tolerance: deployed configuration (model version, env variables, IaC outputs) checked against the DR-approved baseline. Tolerances defined per-control (model patch versions within the same major version are tolerated; model family changes are not tolerated without a DR re-review).

Attestation artifacts are machine-readable, signed, and stored in the SM inventory record. They are regulator-consumable for EU AI Act Art. 9 risk-management evidence and ISO/IEC 42001 AIMS operational records.

Drift opens an IM-Software ticket automatically; the ticket carries the drift dimension (pattern compliance / evidence freshness / configuration), the specific control that failed tolerance, and a link to the DR decision record.

B) Contribute per-archetype configuration baseline schemas

Publish per-archetype IR configuration baseline schemas, defining what "correct" implementation looks like for each AI/HAI software archetype at each SM tier, to: - OpenSSF AI working group, reference attestation schema for AI software implementations; machine-readable format compatible with supply-chain attestation frameworks (SLSA, in-toto). - OWASP SAMM AI extensions, Verification function, Implementation Review stream; practitioner-level checklist items and evidence-type definitions. - CSA AI Safety Initiative AI Controls Matrix, per-archetype configuration controls mapped to CSA control categories.

Internal practice remains aligned to the published external versions; internal-only deviations are proposed as upstream changes.

Adoption tracked: citations, forks, direct acknowledgment from peer organizations, inclusion in external tooling or assessment frameworks.

C) Automated drift-to-IM escalation and SLA enforcement

• All IR findings (whether from daily attestation or from periodic reviews) flow into IM-Software automatically with severity and SLA pre-populated from the SM L2 tier-treatment matrix.
• IM-Software SLA clock starts when the finding is opened; overdue Critical findings escalate to the program sponsor automatically at 50% and 100% of the SLA window.
• Post-incident reviews in IM-Software that touch a configuration or code control automatically re-examine the IR record for the affected artifact, was the drift detectable earlier? What attestation rule would have caught it? The answer updates the attestation rule and the IR checklist.

Outcome Metrics (L3)

Process Metrics (leading)

• Attestation-pipeline health monitored, % Critical artifacts producing a fresh attestation signal in the last 24 hours; on-call paged if any Critical artifact silent for >24 hours.
• Schema publication pipeline, at least one schema in-draft, in-review, or published at any time.
• IM escalation automation tested quarterly, confirm Critical-finding auto-escalation fires correctly for a synthetic finding.
• Post-incident IR feedback loop active, % of IM post-incident reviews that produce an attestation rule update (expected: ≥1 per material incident).

Effectiveness Metrics (business value)

• IR reviewer-hours per artifact trending down as attestation absorbs routine checks.
• Zero Critical-tier go-live events where the DR-approved design and the deployed configuration are materially different, attestation enforces what was once a periodic audit.
• Auditor configuration-claim findings approaching zero as daily attestation corroborates internal records; regulator inquiries answered via machine-readable attestation artifacts without manual assembly.
• External recognition, configuration baseline schemas cited by peer organizations, security tooling vendors, or regulatory guidance documents.

Success Criteria

• Daily attestation operating for ≥90% of Critical-tier artifacts across all three dimensions (pattern compliance, evidence freshness, configuration tolerance); deviations auto-opening IM tickets within 1 hour.
• Zero stale-evidence violations for Critical-tier REMs; High-tier stale-evidence rate trending down.
• Per-archetype configuration baseline schemas published to OpenSSF AI / OWASP SAMM AI / CSA with documented external adoption.
• IR reviewer-hours per Critical artifact per year trending down over two consecutive quarters.

Key Success Indicators

Level 1: - Per-archetype IR checklists published, one per SM-Software archetype (LLM-integrated app, agent, RAG, fine-tune/training, eval harness, model-serving service, classical ML), covering code-matches-pattern verification, config-matches-DR verification, SR REM evidence currency check, logging-event production verification, and kill-switch test; agent checklist verifies tool allowlist enforcement in code, per-tool scope, HITL gate function, and tool-call logging. - Go-live, annual, and material-change review triggers wired to the SM inventory; 100% of new AI/HAI software artifacts in the last 90 days have a go-live IR record; ≥90% of active artifacts carry a current-year IR record. - All Critical / blocker findings resolved before production cutover; High findings closed within 7 days with evidence linked; findings-aging dashboard reviewed monthly by the program sponsor. - SR REM update loop active, IR findings that reveal stale or inaccurate REM evidence trigger REM row updates before the finding is closed.

Level 2: - ≥90% of Critical-tier artifacts under continuous drift detection (code-repo webhooks, model-registry events, IaC scan, deploy-event telemetry, CI/CD parameters); median detection latency ≤7 days. - No-train and retention settings verified via vendor admin APIs (OpenAI / Anthropic / Bedrock / Vertex) for ≥80% of Critical/High-tier artifacts on a monthly (Critical) and quarterly (High) probing cadence, not from DPA text alone. - 100% of Critical/High-tier agent artifacts with tool-scope boundary tests (out-of-scope argument rejection, non-allowlisted tool call blocked, HITL gate trigger verified, kill-switch SLA confirmed) on record in the current IR cycle. - Tier-cadence adherence ≥95%: Critical on semi-annual + continuous, High on annual, Medium on annual, Low on go-live + material-change.

Level 3: - ≥90% of Critical-tier artifacts producing a daily attestation signal across all three dimensions (pattern compliance, evidence freshness, configuration tolerance); deviations auto-opening IM tickets within 1 hour. - Zero stale-evidence violations for Critical-tier REMs; evidence freshness windows enforced per the attestation pipeline. - Per-archetype configuration baseline schemas published to OpenSSF AI, OWASP SAMM AI, or CSA AI Safety Initiative with documented adoption. - IR reviewer-hours per Critical artifact per year trending down over two consecutive quarters as attestation absorbs routine checks.

Common Pitfalls

Level 1: - ❌ IR treated as a one-time go-live formality, no annual re-review and no material-change trigger; configuration drift accumulates silently for quarters until an audit or an incident surfaces it. - ❌ Reviewers take the DR decision record at face value without opening the codebase, the tool allowlist is declared in the checklist but never verified in the deployed code; the allowlist enforcement logic was never wired. - ❌ No-train and retention settings verified from DPA text alone without opening the vendor admin console, the setting can be reset by a vendor product update and the team does not know. - ❌ Kill-switch documented in the DR record but never tested, the IR checklist has a "kill-switch: yes" box that is checked without an actual test execution and a recorded result. - ❌ Finding severity is flat, all findings treated as Medium regardless of impact, so a Critical-level tool-scope expansion sits in the same queue as a logging-format discrepancy. - ❌ Material-change trigger is not wired to SM inventory events, model swaps, new agent tools, and new data classes ship to production without triggering an IR; the review calendar misses the changes entirely.

Level 2: - ❌ Drift-detection pipeline ingests code-repo events but generates no findings on deltas, the pipeline exists but automated finding creation was never configured; drift detection is manual in practice. - ❌ Vendor admin API probing is configured once at onboarding and never re-run, a no-train setting reset by a vendor product update is undetected for months. - ❌ Tool-scope boundary testing is documented as "verified at go-live" but never repeated, agent tool scope may have been widened at a subsequent deploy; the go-live test is the only record. - ❌ Tier-calibrated cadence exists on paper but Critical and Low-tier artifacts sit in the same review queue with no prioritization, Critical-tier artifacts wait behind Low-tier backlogs. - ❌ Drift findings from automated detection dead-end in an alert dashboard rather than auto-opening IM tickets, findings age without owners.

Level 3: - ❌ Daily attestation signals show green across all Critical artifacts but the underlying checks cover only logging volume, tool allowlist enforcement, HITL gate wiring, and model-version tolerance are not checked; attestation is cosmetic. - ❌ Configuration baseline schemas published externally diverge from internal practice, what is published reflects the L1 checklist; internal practice has advanced to L2 tooling and L3 vendor API probing; external adopters build on a stale baseline. - ❌ Attestation-exception queue overwhelms the team because configuration tolerance thresholds are too tight, every dependency version bump triggers a deviation; reviewers suppress the signal source rather than tune the tolerance rules. - ❌ Post-incident IR feedback loop exists in policy but never fires in practice, IM post-incident reviews do not include the IR-record re-examination step; attestation rules never update from incident learning.

Practice Maturity Questions

Level 1: 1. Is there a published, per-archetype IR checklist, one per SM-Software archetype (LLM-integrated app, agent, RAG, fine-tune/training, eval harness, model-serving service, classical ML), covering code-matches-pattern verification, config-matches-DR verification, SR REM evidence currency check, logging-event production verification, and kill-switch test execution, with the agent checklist verifying tool allowlist enforcement in deployed code, per-tool scope, HITL gate function, and tool-call logging? 2. Do 100% of new AI/HAI software artifacts going to production in the last 90 days carry a go-live IR record, and do ≥90% of all active artifacts carry a current-year IR record, with material-change triggers wired to SM inventory events, Critical / blocker findings resolved before production, and High findings closed within 7 days with evidence linked? 3. Are findings severity-tagged and tracked in IM-Software with named owners and SLA-bound closure dates, and does every IR finding that reveals stale or inaccurate REM evidence trigger an SR REM row update before the finding is closed?

Level 2: 1. Are ≥90% of Critical-tier AI/HAI software artifacts under continuous drift detection, via code-repo webhooks, model-registry events, IaC scan tooling, deploy-event telemetry, and CI/CD parameter monitoring, with median detection latency ≤7 days and automated finding creation on material deviations? 2. Are no-train and retention settings verified via vendor admin APIs (OpenAI / Anthropic / Bedrock / Vertex / equivalent) on a monthly (Critical) and quarterly (High) probing cadence, not from DPA text alone, covering ≥80% of Critical/High-tier artifacts, with deltas from the previous probe opening IR findings with severity matching the data-class impact? 3. Are 100% of Critical/High-tier agent artifacts covered by tool-scope boundary tests in the current IR cycle, confirming out-of-scope argument rejection, non-allowlisted tool-call blocking, HITL gate trigger, and kill-switch SLA, and is tier-cadence adherence ≥95% with Critical-tier findings aged per the SM L2 tier-treatment matrix SLAs?

Level 3: 1. Are ≥90% of Critical-tier AI/HAI software artifacts producing a daily attestation signal across all three dimensions (pattern compliance, evidence freshness, configuration tolerance), with deviations auto-opening IM-Software tickets within 1 hour and zero stale-evidence violations for Critical-tier REMs? 2. Has the program published per-archetype configuration baseline schemas to OpenSSF AI, OWASP SAMM AI, or CSA AI Safety Initiative, with documented adoption and internal practice aligned to the published versions, and is IR reviewer-hours per Critical artifact per year trending down over two consecutive quarters? 3. Is the post-incident IR feedback loop operational, IM-Software post-incident reviews include a mandatory IR-record re-examination step, and ≥1 attestation rule update is produced per material incident, ensuring incident learning continuously improves the attestation coverage?

Document Version: HAIAMM v3.0 Practice: Implementation Review (IR) Domain: Software Last Updated: 2026-05-13 Author: Verifhai