Implementation Review (IR)

Vendors Domain - HAIAMM v3.0

Practice Overview

Objective: Verify, at the point of deployment and on a recurring cadence, that the actual AI vendor integration configuration matches the design approved at DR, closing the gap between what was designed and what is running.

Description: IR-Vendors is the configuration check for AI vendor integrations, the moment someone opens the vendor's admin console, the app's config, the proxy rules, and the SaaS admin settings and confirms they match the approved design. At L1 the review is a short checklist per archetype, focused on the handful of settings where production reality most commonly drifts from the design.

Context: AI vendor admin consoles are deep and fast-moving. Defaults change; vendors quietly add new AI features to existing panels; API-key permissions drift; logging toggles revert to off after a product update. Without an implementation review, the "we turned off training" decision from DR becomes "we thought we turned off training" six months later. L1 IR-Vendors makes the config check a regular, lightweight activity rather than a periodic audit scramble.

Maturity Level 1

Objective: Run an implementation review at go-live and at least annually per AI vendor integration, using a per-archetype config checklist; track findings to closure

At this level, configuration is checked, not assumed, at the moments it matters most (go-live, annually, after vendor material changes).

Dependencies

• DR-Vendors L1 (required): the approved design is the spec IR reviews against.
• SR-Vendors L1 (required): the requirements pack provides the items being verified.
• SA-Vendors L1 (required): the reference pattern provides the intended configuration shape.
• EG-Vendors L1 (required): reviewers understand admin consoles specific to each archetype.
• Supports / unblocks: ST-Vendors L1 (tests target verified config), EH-Vendors L1 (hardening acts on IR findings), IM-Vendors L1 (findings become issues in the backlog), ML-Vendors L1 (logging config verified here feeds monitoring).

Desired Outcomes

• The gap between approved design and live configuration stays small and short-lived.
• Material configuration drift gets found by the program, not by an auditor or an incident.
• Findings have named owners and SLAs for closure; aging findings are visible.
• Vendor product changes that silently alter configuration (new default, new feature toggle) are detected on the next cadence, not years later.

Activities

A) Publish the per-archetype implementation review checklist

One short checklist per archetype, focused on settings that matter for SR/TA coverage.

Common spine across archetypes: - No-train / retention settings, verify admin-console toggles match DPA commitments. - SSO / identity binding, verify SSO enforced, local-auth disabled, admin-role separation. - Logging enabled and exported, verify logs cover the required events, retention is policy-compliant, export path works. - API key / token scope, verify least-privilege scope, owner attribution, rotation schedule. - Rate / abuse limits, verify caps align with DR-approved assumptions. - Region / residency, verify region setting matches DPA and residency commitments. - New AI features surfaced, vendor panel scanned for features added since last review; changes flagged to DR.

Archetype-specific emphasis: - Consumer GenAI, org tenant binding, content-filter settings, memory-off if required, audit-log access. - AI-embedded SaaS, per-workspace feature toggle state, users with access, data-type boundaries. - AI coding assistant, IDE-policy enforcement, regulated-repo no-train path, telemetry scope. - AI API / model, proxy presence and health, model-version pin, PII scrubbing active, kill-switch wired. - AI agent / automation platform, tool allowlist, per-tool scope, HITL gates, session logging, untrusted-content provenance handling.

B) Perform reviews at the right moments

Three triggers at L1: - Go-live review, before production cutover, reviewing the as-built config against the DR-approved design. - Annual review, every active AI vendor integration reviewed at least annually; scheduled from the inventory. - Change-triggered review, vendor major product update, pricing-plan change, org-plan migration, or material admin-console redesign triggers an ad-hoc review.

Reviews are short (target 20–45 minutes per integration) and evidence-based, screenshots or config exports stored with the record.

C) Track findings to closure

Every review produces zero or more findings. Findings get: - Severity, blocker / high / medium / low. - Owner, named integration owner or vendor-admin owner. - SLA, blocker fix before production (or rollback); high ≤14 days; medium ≤45 days; low ≤90 days or accepted residual. - Evidence, after-fix screenshot / config export linked to close.

Findings feed IM-Vendors as issues (for tracking, aging, trending) and EH-Vendors as hardening work where the pattern itself needs changes.

Outcome Metrics (L1)

Process Metrics (leading)

• Annual review calendar populated from the inventory; integrations nearing review-due date visible in advance.
• Change-triggered reviews queued within 5 business days of a detected vendor change.
• Reviewer backlog aging, no single reviewer >3 integrations overdue.

Effectiveness Metrics (business value)

• Drift-caught-early rate, findings closed before they reach ST or an incident.
• Pattern-feedback rate, findings triggering SA pattern updates.
• Avoided-incident stories, documented cases where IR caught a config regression before exploit/exposure.

Success Criteria

• Per-archetype checklists published and owned.
• Go-live, annual, and change-triggered review triggers wired to the inventory and integration tracker.
• 100% of new AI vendor integrations in the last 90 days have a go-live IR record.
• ≥90% of active AI vendor integrations carry a current-year IR record.
• Findings-aging dashboard reviewed at least monthly by the program sponsor.

Maturity Level 2

Objective: Continuous configuration validation via vendor APIs; automated no-train and retention verification; tier-calibrated IR cadence

At this level, implementation review stops being a point-in-time check and becomes a continuous signal. Critical-tier vendors' admin APIs are consumed by the program to detect config drift in real time; no-train and retention settings are validated via recurring probes; cadence is tier-aware.

Dependencies

• IR-Vendors L1 (required): per-archetype checklists and findings workflow.
• SM-Vendors L2 (required): tiers drive cadence and depth.
• SA-Vendors L2 (required): IaC-encoded patterns establish the "correct" baseline.
• Supports / unblocks: ST-Vendors L2 (tests run against verified config), EH-Vendors L2, ML-Vendors L2.

Desired Outcomes

• Config drift on Critical-tier integrations detected within days, not months.
• No-train and retention settings validated via observable signal, not only by DPA text.
• IR cadence differentiates clearly by tier; Low-tier isn't flooded with the same depth as Critical.

Activities

A) API-based config monitoring

• Critical vendors' admin APIs consumed by the program to produce a live config snapshot.
• Change events flow into IR queue; material changes auto-generate findings.
• Where vendor APIs don't exist, scheduled UI-scraping or attestation confirmations fill in.

B) Automated no-train and retention verification

• Recurring synthetic canaries for vendors where probing is contract-permitted.
• Admin-console state polled; deltas vs. DPA commitments raise findings.
• Retention-window sampling: prompt/completion logs older than agreed retention trigger evidence request.

C) Tier-calibrated cadence

• Critical: go-live + semi-annual + change-triggered + continuous drift detection.
• High: go-live + annual + change-triggered.
• Medium: go-live + annual.
• Low: go-live only + re-review on change.

Outcome Metrics (L2)

Process Metrics (leading)

• API-ingestion pipeline health monitored.
• Canary probing on calendar for eligible vendors.
• IR backlog tier-aware; aging reported per tier.

Effectiveness Metrics (business value)

• Drift caught before ST/ML/incidents, trend.
• Reduced auditor findings on configuration claims.

Success Criteria

• ≥90% Critical vendors under API-based config monitoring.
• Critical drift detection ≤7 days.
• Automated no-train verification for ≥80% Critical.
• Tier-cadence adherence ≥95%.

Maturity Level 3

Objective: Real-time configuration attestation; contribute config-baseline schemas to industry; integrate with vendor trust-center attestations

At this level, configuration isn't "reviewed", it's attested continuously. Config-baseline schemas (what "correct" looks like per archetype) are published industry artifacts. Vendor trust centers (SIG, CAIQ, custom) feed into the program's attestation.

Dependencies

• IR-Vendors L2 (required): API-based monitoring, automated verification, tier cadence.
• SA-Vendors L3 (required): externalized patterns.

Desired Outcomes

• Config state readable in near-real-time for Critical-tier integrations.
• Config-baseline schemas cited externally.
• Trust-center integrations (SIG, CAIQ, vendor-specific) reduce duplicate evidence collection.

Activities

A) Real-time attestation

• Config state evaluated continuously; any deviation raises an instant finding.
• Attestation artifacts machine-readable and regulator-consumable.

B) Contribute config-baseline schemas

• Per-archetype IR config-baseline schemas (what "correct" looks like) published to Shared Assessments, CSA AI Safety Initiative.

C) Trust-center integration

• Ingest vendor trust-center artifacts (SIG Lite, CAIQ, custom) into IR evidence automatically.
• Deltas from DPA / AI addendum auto-flagged.

Outcome Metrics (L3)

Process Metrics (leading)

• Attestation-pipeline health monitored, % Critical vendors producing a fresh attestation signal within the last 24 hours.
• Config-schema publication pipeline, at least one schema in-draft, in-review, or published at any time.
• Trust-center ingestion backlog, new vendors onboarded to the feed within 30 days of identification.
• Attestation-exception queue, deviations from continuous attestation triaged within 1 business day.

Effectiveness Metrics (business value)

• IR-reviewer hours per integration trending down quarter-over-quarter as attestation absorbs routine checks.
• Auditor configuration-claim findings approaching zero as trust-center feeds corroborate internal records.
• Regulator or standards-body citation of published config-baseline schemas signals industry recognition.
• Vendor-cooperation depth increased, trust-center integrations reduce evidence-collection burden for new regulatory requests.

Success Criteria

• Real-time attestation for ≥90% Critical.
• Config-baseline schemas published with external adoption.
• Trust-center feeds live for ≥70% Critical/High.

Key Success Indicators

Level 1: - Per-archetype IR checklists published, covering no-train/retention toggles, SSO binding, logging export, API-key scope, region residency, and new-feature surfacing, linked from the integration tracker. - Go-live, annual, and change-triggered review cadences wired to the inventory; 100% of new AI vendor integrations in the last 90 days carry a go-live IR record. - ≥90% of active AI vendor integrations carry a current-year IR record; findings-aging dashboard reviewed monthly by the program sponsor. - All blocker findings resolved before production cutover; high findings closed within 14 days with evidence linked.

Level 2: - ≥90% of Critical-tier vendors under API-based config monitoring; median drift detection time ≤7 days. - Automated no-train and retention verification active for ≥80% of Critical-tier vendors, polling admin-console state and comparing against DPA commitments, not trusting DPA text alone. - Tier-cadence adherence ≥95%: Critical on semi-annual + continuous, High on annual, Medium on annual, Low on go-live + change-triggered. - IR backlog reported per tier; Critical-tier findings never wait behind Low-tier queue items.

Level 3: - ≥90% of Critical-tier integrations produce real-time config attestation signals; deviations open exception tickets within 1 business day. - Per-archetype config-baseline schemas (defining what "correct" looks like) published to Shared Assessments or CSA AI Safety Initiative, with external adoption tracked. - Trust-center feeds (SIG Lite, CAIQ, vendor-custom) ingested for ≥70% of Critical/High vendors; DPA/AI-addendum deltas auto-flagged.

Common Pitfalls

Level 1: - ❌ IR treated as a one-time go-live formality, no annual re-review, so configuration drift accumulates silently for years. - ❌ Reviewers take vendor's DPA text at face value for no-train and retention settings without opening the admin console to verify the toggle state. - ❌ Checklists focus on classic SaaS settings and miss AI-specific configuration points: model-version pins, prompt/completion log export paths, HITL-gate wiring, agent tool-allowlist state. - ❌ Finding severity is flat, all findings treated as medium regardless of security impact, so blockers sit in the same queue as cosmetic items. - ❌ Change-triggered reviews never fire because the trigger isn't wired to vendor-change notifications; reviews run only on the annual calendar.

Level 2: - ❌ API-based monitoring ingests config snapshots but generates no alerts on deltas, the pipeline exists but drift detection is manual. - ❌ Automated no-train verification runs once at onboarding and is never re-run; the setting can change between checks without detection. - ❌ Tier calibration exists on paper but the IR team applies Critical-level depth to every vendor anyway, no actual differentiation in cadence or tooling. - ❌ IR backlog per tier is reported but not acted on, Critical-tier aging findings treated identically to Low-tier. - ❌ Drift findings from API monitoring dead-end in a spreadsheet rather than auto-opening IM tickets.

Level 3: - ❌ Real-time attestation becomes telemetry theater, dashboard shows green across all Critical vendors but the underlying checks don't cover the configuration settings that matter. - ❌ Config-baseline schemas published externally diverge from internal practice, what's published is a historical snapshot, not the living standard reviewers use. - ❌ Trust-center ingestion pipeline built but not maintained, vendor attestation feeds go stale when vendors rotate their publishing cadence. - ❌ Attestation-exception queue overwhelms the team because thresholds are too sensitive; reviewers suppress alerts rather than tune them.

Practice Maturity Questions

Level 1: 1. Is there a published, per-archetype IR checklist covering no-train/retention toggle verification, SSO binding, logging export confirmation, API-key scope, region residency, and new-feature surfacing, with named reviewers per archetype and target timebox of 20–45 minutes per integration? 2. Do 100% of new AI vendor integrations going to production in the last 90 days carry a go-live IR record, and do ≥90% of all active integrations carry a current-year review record, with blocker findings resolved before production and high findings closed within 14 days? 3. Are findings severity-tagged and tracked in IM with named owners and SLA-bound closure dates, and is the findings-aging dashboard reviewed at least monthly by the program sponsor?

Level 2: 1. Are ≥90% of Critical-tier vendors under API-based config monitoring that detects configuration drift within ≤7 days of a change, and does the monitoring auto-generate IM findings on material deltas, not rely on manual review? 2. Is no-train and retention verification conducted via recurring admin-console state polling and canary probes (where contract-permitted) rather than trusting DPA text alone, covering ≥80% of Critical-tier vendors? 3. Is tier-cadence adherence ≥95%, Critical on semi-annual + continuous drift, High on annual, Medium on annual, Low on go-live + change-triggered, with the IR backlog reported and aged per tier?

Level 3: 1. Are ≥90% of Critical-tier AI vendor integrations producing real-time config attestation signals, with attestation deviations auto-opening IR exception tickets triaged within 1 business day? 2. Has the program published per-archetype config-baseline schemas (defining "correct" configuration for each AI-vendor archetype) to an external body (Shared Assessments, CSA AI Safety Initiative) with documented adoption? 3. Are trust-center feeds (SIG Lite, CAIQ, vendor-custom) ingested for ≥70% of Critical/High vendors, with DPA and AI-addendum deltas auto-flagged rather than discovered at annual review?

Document Version: HAIAMM v3.0 Practice: Implementation Review (IR) Domain: Vendors Last Updated: 2026-05-12 Author: Verifhai