Implementation Review (IR) - Vendors Assessment

Assessment questionnaire for measuring maturity. Answer each question honestly based on current, implemented practices.

Implementation Review (IR) - Vendors Domain

HAIAMM Assessment Questionnaire v3.0

Practice: Implementation Review (IR) Domain: Vendors Purpose: Assess organizational maturity in verifying that the actual AI vendor integration configuration matches the design approved at DR, closing the gap between what was designed and what is running. Scoring Model: Evidence + Outcome Metrics (see Scoring Methodology below)

Instructions

Answer each question honestly based on current, implemented practices (not plans or aspirations)
Each question has two components: Evidence (what you did) and Outcome Metrics (how well it worked)
Scoring uses 4 tiers: Fully Mature (1.0), Implemented (0.67), Partial (0.33), Not Implemented (0.0)
Answer progressively, Complete all Level 1 questions before Level 2
Level progression, Achieve ALL questions at lower level before advancing
Baseline first, Record current metric values before setting targets

Scoring Methodology

Score	Label	Criteria
1.0	Fully Mature	Evidence complete + ≥3 outcome metrics meet targets
0.67	Implemented	Evidence complete + 2 outcome metrics meet targets
0.33	Partial	Evidence partially complete + <2 metrics meet targets
0.0	Not Implemented	No evidence of practice

Level Score = Average of the three question scores at that level Overall Score = Weighted average across all levels achieved

Maturity Level 1

Objective: Run an implementation review at go-live and at least annually per AI vendor integration, using a per-archetype config checklist; track findings to closure

At this level, configuration is checked, not assumed, at the moments it matters most (go-live, annually, after vendor material changes).

Question 1: Per-Archetype Vendor IR Checklist

Q1.1: Is there a published, per-archetype IR checklist, one per AI vendor archetype (consumer GenAI, AI-embedded SaaS, AI coding assistant, AI API / model, AI agent / automation platform), covering no-train/retention toggle verification, SSO binding, logging export, API-key scope, region residency, and new-feature surfacing?

Evidence Required: - [ ] Published checklist per AI vendor archetype on file and linked from the integration tracker (consumer GenAI, AI-embedded SaaS, AI coding assistant, AI API / model, AI agent / automation platform) - [ ] No-train / retention toggle verification present in each checklist: reviewer opens the vendor admin console and confirms the toggle state, not accepting DPA text as evidence of the actual toggle position - [ ] SSO / identity binding verified: SSO enforced, local-auth disabled, admin-role separation confirmed via the admin settings panel or admin API - [ ] Logging enabled and exported: logs cover required events, retention is policy-compliant, export path to SIEM confirmed working; evidence: a sample log export on file with the IR record - [ ] API key / token scope verified: least-privilege scope confirmed, owner attribution present, rotation schedule on file; keys audited against the DR-approved scope - [ ] New AI features surfaced: vendor panel scanned for features added or changed since the last review; new features flagged to DR for a design decision before enabling

Outcome Metrics:

Metric	Baseline	Current	Target	Met?	Notes
% AI vendor integrations with a go-live IR record	___%	___%	100%	☐	Integration tracker
% active AI vendor integrations with a current-year IR record	___%	___%	≥90%	☐	Inventory × IR records
Blocker findings open at go-live	___	___	0	☐	Findings backlog
Median closure time for High findings	___ days	___ days	≤14 days	☐	Findings backlog

Metric Collection Guidance: - Go-live IR coverage: count integrations in the integration tracker with a linked IR record dated at or before production cutover divided by total integrations entering production. Formula: integrations_with_golive_IR / total_integrations_in_production × 100 - Current-year IR coverage: count integrations with an IR record dated within the last 12 months divided by total active integrations. Source: inventory last-IR-date field - Blocker findings at go-live: count blocker findings that were open on the go-live date. Source: IM-Vendors findings history - High finding closure time: median calendar days from finding-opened to evidence-linked finding-closed for High-severity findings. Source: IM-Vendors backlog timestamps

Answer: - ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets) - ☐ Implemented (Evidence complete + 2 metrics meet targets) - ☐ Partial (Evidence partially complete + <2 metrics meet targets) - ☐ Not Implemented (No per-archetype checklist published)

Evidence Location: _________

Validation Date: _________

Notes: _______

Question 2: Review Triggers and Timing

Q1.2: Do 100% of new AI vendor integrations going to production in the last 90 days carry a go-live IR record, and do ≥90% of all active integrations carry a current-year IR record, with change-triggered reviews wired to vendor-change notifications?

Evidence Required: - [ ] Go-live IR records on file for all integrations entering production in the last 90 days; screenshots or config exports confirm the IR was completed before the production cutover (target timebox 20–45 minutes per integration) - [ ] Change-triggered reviews wired to vendor-change notifications: vendor major product updates, pricing-plan changes, org-plan migrations, and material admin-console redesigns generate a review-due alert within 5 business days of detection - [ ] SaaS admin audit feeds reviewed at each change-triggered IR: M365 admin audit log, Slack admin event log, Google Workspace admin audit, Notion admin settings reviewed for AI feature enable/disable events and data-scope configuration changes since the last IR - [ ] Region / residency setting verified: region setting in vendor admin console confirmed to match DPA and residency commitments declared in the SR-Vendors REM; evidence: admin console screenshot on file - [ ] Rate / abuse limits verified: caps confirmed to align with DR-approved assumptions; evidence: admin console rate-limit setting export or screenshot - [ ] Annual review calendar populated from the inventory with integrations nearing review-due date visible at least 30 days in advance

Outcome Metrics:

Metric	Baseline	Current	Target	Met?	Notes
% AI vendor integrations with a go-live IR record	___%	___%	100%	☐	Integration tracker
% active AI vendor integrations with a current-year IR record	___%	___%	≥90%	☐	Inventory × IR records
Blocker findings open at go-live	___	___	0	☐	Findings backlog
% reviews that surfaced at least one material finding	___%	___%	tracked as a trend	☐	Findings data

Metric Collection Guidance: - Current-year IR coverage: count integrations with an IR record dated within the last 365 days divided by total active integrations. Source: inventory last-IR-date - Change-triggered review rate: cross-reference vendor-change notifications against change-triggered IR records created within 5 business days of each event - Material finding rate: count reviews that surfaced at least one finding with severity High or Critical divided by total reviews completed. Source: IM-Vendors findings history - Reviewer backlog aging: no single reviewer more than 3 integrations overdue. Source: IR assignment log

Evidence Location: _________

Validation Date: _________

Notes: _______

Question 3: Findings Tracking and Closure

Q1.3: Are findings severity-tagged and tracked in IM-Vendors with named owners and SLA-bound closure dates, and is the findings-aging dashboard reviewed at least monthly by the program sponsor?

Evidence Required: - [ ] Findings backlog in IM-Vendors showing all IR findings with severity tag (blocker / high / medium / low), named owner (named integration owner or vendor-admin owner, not "the IT team"), SLA-bound closure date, and linked after-fix evidence artifact (screenshot or config export) - [ ] SLA applied per severity: blocker resolved before production cutover or rollback; high ≤14 days; medium ≤45 days; low ≤90 days or accepted residual - [ ] Findings feed to EH-Vendors as hardening work where the pattern itself needs changes: at least one finding that triggered a pattern update documented in the last 12 months - [ ] Severity calibration consistent: blocker findings include AI-specific configuration points (model-version pin missing, prompt/completion log export path broken, HITL-gate not wired for AI agent integrations, agent tool-allowlist not verified), not treated as medium - [ ] Findings-aging dashboard reviewed at least monthly by the program sponsor: meeting record or dashboard screenshot on file - [ ] Contract change events and subprocessor change notifications reviewed at each IR: any vendor contract change since the last IR flagged as a material change and reviewed for configuration impact

Outcome Metrics:

Metric	Baseline	Current	Target	Met?	Notes
% AI vendor integrations with a go-live IR record	___%	___%	100%	☐	Integration tracker
% active AI vendor integrations with a current-year IR record	___%	___%	≥90%	☐	Inventory × IR records
Blocker findings open at go-live	___	___	0	☐	Findings backlog
Median closure time for High findings	___ days	___ days	≤14 days	☐	Findings backlog

Metric Collection Guidance: - Named-owner coverage: count findings with a named individual as owner divided by total open findings. Source: IM-Vendors - SLA adherence by severity: % of high findings closed within 14 days; % of blocker findings resolved before go-live. Source: IM-Vendors timestamps - Pattern-feedback rate: count findings that triggered an SA-Vendors pattern update divided by total findings closed in the period. Source: IM-Vendors × SA-Vendors change log - Findings-aging review cadence: confirm monthly review via calendar record or dashboard export. Source: program sponsor review log

Evidence Location: _________

Validation Date: _________

Notes: _______

Maturity Level 2

Objective: Continuous configuration validation via vendor APIs; automated no-train and retention verification; tier-calibrated IR cadence

At this level, implementation review becomes a continuous signal for Critical-tier vendors. Vendor admin APIs are consumed to detect config drift; no-train and retention settings are validated via recurring probes; cadence is tier-aware.

Question 4: API-Based Config Monitoring for Critical Vendors

Q2.1: Are ≥90% of Critical-tier vendors under API-based config monitoring that detects configuration drift within ≤7 days of a change, and does the monitoring auto-generate IM-Vendors findings on material deltas?

Evidence Required: - [ ] Critical vendors' admin APIs consumed by the program to produce a live config snapshot: API ingestion pipeline configuration on file; the pipeline covers no-train/retention settings, SSO binding state, logging configuration, API-key scope, and rate-limit settings - [ ] Change events from vendor APIs flow into the IR queue: material changes auto-generate findings in IM-Vendors with severity and owner pre-populated, not dead-ending in a spreadsheet or dashboard - [ ] SaaS admin audit feed ingestion active: M365 / Slack / Workspace / Notion admin event webhooks configured to flag AI feature enablement events, data-scope configuration changes, and permission changes; ingestion pipeline health monitored - [ ] Where vendor APIs do not exist, scheduled UI-scraping or attestation confirmations documented as a gap with a plan and timeline for migration to API-based verification - [ ] Detection latency evidence on file: telemetry log showing Critical-tier vendor config drift findings opened within 7 days of the change event for the last 90-day period - [ ] API-ingestion pipeline health monitored: % Critical vendors producing a fresh config snapshot in the last 7 days; on-call alert configured and tested for >48 hours of feed silence

Outcome Metrics:

Metric	Baseline	Current	Target	Met?	Notes
% Critical vendors with API-based config monitoring live	___%	___%	≥90%	☐	Monitoring telemetry
Median detection time for Critical config drift	___ days	___ days	≤7 days	☐	IR telemetry
% Critical vendors with automated no-train and retention verification	___%	___%	≥80%	☐	Verification telemetry
Tier-cadence adherence	___%	___%	≥95%	☐	IR schedule

Metric Collection Guidance: - API-based monitoring coverage: count Critical-tier vendors with an active API ingestion pipeline divided by total Critical-tier vendors. Source: monitoring pipeline configuration registry - Median detection latency: median time from vendor-side change-event to IM-Vendors finding opened. Source: IR telemetry × IM-Vendors created-at timestamps - Automated no-train verification coverage: count Critical-tier vendors with automated no-train and retention verification (admin-console state polling or admin API probe) divided by total Critical-tier vendors. Source: verification telemetry - Tier-cadence adherence: % of integrations reviewed within their published cadence window (Critical: semi-annual + continuous; High: annual). Source: IR schedule × inventory last-IR-date

Evidence Location: _________

Validation Date: _________

Notes: _______

Question 5: Automated No-Train and Retention Verification

Q2.2: Is no-train and retention verification conducted via recurring admin-console state polling and vendor admin API probes (where contract-permitted) rather than trusting DPA text alone, covering ≥80% of Critical-tier vendors?

Evidence Required: - [ ] Vendor admin API probe records on file for Critical-tier vendors using LLM providers: OpenAI Org Settings API (data_controls.training_data_sharing = false); Anthropic Organization admin settings confirming no-train terms; Bedrock Service Control Policy and invocation logging config; Vertex AI / Gemini Organization Policy constraints, polled recurrently, not only at onboarding - [ ] Admin-console state polled on a schedule: Critical-tier vendors polled monthly; High-tier vendors polled quarterly; any delta vs. DPA commitments raises a finding in IM-Vendors - [ ] Retention-window sampling active: prompt/completion logs older than the agreed retention window verified as purged or trigger an evidence request; sampling cadence documented - [ ] Contract change events reviewed: subprocessor change notifications and AI addendum changes flagged and reviewed for impact on no-train and retention settings; evidence of a process for receiving and acting on such notifications - [ ] Delta findings on file: any setting change detected (no-train toggle reset, retention period changed, subprocessor changed) generated a finding with severity matching the data-class impact - [ ] Probing calendar maintained: missed probes tracked as process-metric failures with root cause and remediation on file

Outcome Metrics:

Metric	Baseline	Current	Target	Met?	Notes
% Critical vendors with API-based config monitoring live	___%	___%	≥90%	☐	Monitoring telemetry
Median detection time for Critical config drift	___ days	___ days	≤7 days	☐	IR telemetry
% Critical vendors with automated no-train and retention verification	___%	___%	≥80%	☐	Verification telemetry
Tier-cadence adherence	___%	___%	≥95%	☐	IR schedule

Metric Collection Guidance: - Automated no-train verification coverage: count Critical-tier vendors with a vendor admin API probe result dated within the current probe cycle divided by total Critical-tier vendors. Source: verification telemetry - Delta finding rate: count probe cycles where a setting change was detected and a finding was generated vs. total probe cycles. Source: probe log × IM-Vendors - Retention-window sampling coverage: % of Critical-tier vendors with a retention-window sampling check completed in the current period. Source: IR records - Subprocessor change review rate: % of subprocessor change notifications that triggered a review within 5 business days. Source: vendor notification log × IR records

Evidence Location: _________

Validation Date: _________

Notes: _______

Question 6: Tier-Calibrated Vendor IR Cadence

Q2.3: Is tier-cadence adherence ≥95%, Critical on semi-annual + continuous drift, High on annual, Medium on annual, Low on go-live + change-triggered, with the IR backlog reported and aged per tier?

Evidence Required: - [ ] Tier-cadence policy documented and published: Critical, go-live + semi-annual + change-triggered + continuous drift detection; High, go-live + annual + change-triggered; Medium, go-live + annual; Low, go-live + re-review on change - [ ] IR schedule in the inventory confirmed against policy: every integration carries a last-IR-date and next-IR-due field; Critical-tier integrations with no IR in the last 180 days flagged for escalation to the program sponsor - [ ] IR backlog per tier reported separately: Critical-tier findings aged and reported to the program sponsor with a different priority queue from Low-tier findings; evidence: tiered backlog dashboard or report on file - [ ] Critical-tier findings never wait behind Low-tier queue items: queue-management practice documented and confirmed via a sample review of the IR assignment log in the last 90 days - [ ] Tier-SLA breach rate tracked: count Critical-tier findings exceeding the SLA divided by total Critical-tier findings; rate reported to program sponsor monthly - [ ] Drift findings from API monitoring auto-open IM-Vendors tickets, not dead-ending in an alert dashboard; confirmed via a sample of auto-opened findings from the last 90 days

Outcome Metrics:

Metric	Baseline	Current	Target	Met?	Notes
% Critical vendors with API-based config monitoring live	___%	___%	≥90%	☐	Monitoring telemetry
Median detection time for Critical config drift	___ days	___ days	≤7 days	☐	IR telemetry
% Critical vendors with automated no-train and retention verification	___%	___%	≥80%	☐	Verification telemetry
Tier-cadence adherence	___%	___%	≥95%	☐	IR schedule

Metric Collection Guidance: - Tier-cadence adherence: count integrations reviewed within their published cadence window divided by total integrations. Source: IR schedule × inventory last-IR-date - Critical-tier SLA breach rate: count Critical-tier findings exceeding the tier-treatment matrix SLA divided by total Critical-tier findings. Source: IM-Vendors timestamps - Auto-open ticket rate: count drift findings that auto-opened an IM-Vendors ticket within 24 hours divided by total drift findings. Source: API monitoring telemetry × IM-Vendors - Backlog-aging by tier: median age of open findings by tier. Source: IM-Vendors tiered aging report

Evidence Location: _________

Validation Date: _________

Notes: _______

Maturity Level 3

Objective: Real-time configuration attestation; contribute config-baseline schemas to industry; integrate with vendor trust-center attestations

At this level, configuration is attested continuously for Critical-tier integrations. Per-archetype config-baseline schemas are published externally. Vendor trust-center feeds (SIG Lite, CAIQ, vendor-custom) are ingested into the program's attestation pipeline.

Question 7: Real-Time Configuration Attestation

Q3.1: Are ≥90% of Critical-tier AI vendor integrations producing real-time config attestation signals, with attestation deviations auto-opening IR exception tickets triaged within 1 business day?

Evidence Required: - [ ] Real-time attestation pipeline operational: config state evaluated continuously for Critical-tier integrations; any deviation (no-train toggle changed, SSO enforcement removed, logging export path broken, API-key scope widened, rate-limit cap removed, region setting changed) raises an instant finding - [ ] Attestation artifacts machine-readable, signed, and stored in the integration inventory record: format confirmed suitable for regulatory consumption (EU AI Act Art. 9 risk-management evidence, deployer-duty records per Art. 26) via test export - [ ] Attestation deviations auto-opening IR exception tickets in IM-Vendors within 1 business day: ticket carries the specific setting that changed, the expected value, the observed value, and a link to the DR-approved design - [ ] Attestation-pipeline health monitored: % Critical vendors producing a fresh attestation signal in the last 24 hours; on-call paged if any Critical vendor silent for >24 hours - [ ] Attestation-exception queue managed: deviations triaged within 1 business day; queue depth and aging reported; thresholds tuned to avoid alert fatigue from minor non-material changes - [ ] Zero Critical-tier integrations with a config state not readable via the attestation pipeline: any integration lacking API coverage has a documented gap and a migration-to-API plan on file

Outcome Metrics:

Metric	Baseline	Current	Target	Met?	Notes
% Critical vendors with real-time attestation	___%	___%	≥90%	☐	Attestation telemetry
External adoption of config schemas	0	tracked	trending up	☐	External telemetry
% vendors with trust-center feed ingested	___%	___%	≥70% Critical/High	☐	Integration registry
IR reviewer-hours per integration trending down	___ hrs	___ hrs	trending down QoQ	☐	Reviewer time tracking

Metric Collection Guidance: - Real-time attestation coverage: count Critical-tier integrations with an attestation signal dated within the last 24 hours divided by total Critical-tier integrations. Source: attestation telemetry - Exception ticket latency: median time from attestation deviation detected to IM-Vendors exception ticket opened. Source: attestation pipeline × IM-Vendors created-at timestamps - Attestation pipeline health: % Critical vendors with a fresh signal in the last 24 hours. Source: attestation pipeline health monitor - IR reviewer-hours: total hours logged for Critical-tier vendor IR activities per year. Source: reviewer time tracking

Evidence Location: _________

Validation Date: _________

Notes: _______

Question 8: Contribute Vendor Config-Baseline Schemas

Q3.2: Has the program published per-archetype config-baseline schemas (defining "correct" configuration for each AI-vendor archetype) to an external body, with documented external adoption?

Evidence Required: - [ ] Per-archetype IR config-baseline schemas published to at least one external body: Shared Assessments (SIG), CSA AI Safety Initiative, or equivalent; schemas define what "correct" implementation looks like for each AI-vendor archetype (consumer GenAI, AI-embedded SaaS, AI coding assistant, AI API / model, AI agent / automation platform) - [ ] Internal practice aligned to published external versions: documented mapping between internal IR checklist and published schema; internal-only deviations proposed as upstream changes with a PR or issue link - [ ] External adoption tracked: citations, forks, direct acknowledgment from peer organizations, or inclusion in external tooling or assessment frameworks, evidence on file - [ ] Schema publication pipeline active: at least one schema in-draft, in-review, or published at any time; publication status tracker maintained - [ ] Adoption trend documented: at least one adoption data point on file since initial publication - [ ] Internal schema version pinned to published external version: divergence documented as a proposed upstream change

Outcome Metrics:

Metric	Baseline	Current	Target	Met?	Notes
% Critical vendors with real-time attestation	___%	___%	≥90%	☐	Attestation telemetry
External adoption of config schemas	0	tracked	trending up	☐	External telemetry
% vendors with trust-center feed ingested	___%	___%	≥70% Critical/High	☐	Integration registry
IR reviewer-hours per integration trending down	___ hrs	___ hrs	trending down QoQ	☐	Reviewer time tracking

Metric Collection Guidance: - External adoption: count citations, forks, and direct acknowledgments per quarter since publication. Source: GitHub stars/forks, external publication references, peer organization acknowledgments - Schema alignment rate: % of internal IR checklist items directly traceable to a published external schema item. Source: internal-to-external schema mapping document - Publication pipeline health: number of schemas in active publication stages. Source: publication tracker

Evidence Location: _________

Validation Date: _________

Notes: _______

Question 9: Trust-Center Integration and Vendor Cooperation

Q3.3: Are trust-center feeds (SIG Lite, CAIQ, vendor-custom) ingested for ≥70% of Critical/High vendors, with DPA and AI-addendum deltas auto-flagged rather than discovered at annual review?

Evidence Required: - [ ] Trust-center ingestion pipeline operational: SIG Lite, CAIQ, and vendor-custom attestation feeds ingested into the IR evidence pipeline for Critical and High-tier vendors; ingestion registry on file showing coverage per vendor - [ ] DPA / AI-addendum delta auto-flagging active: any change in the vendor's trust-center artifact (new subprocessor, changed data-processing terms, changed AI training terms) auto-flagged as a potential IR finding for review; at least one example of an auto-flagged delta in the last 12 months on file - [ ] New vendors onboarded to the trust-center feed within 30 days of identification: ingestion backlog tracked; onboarding SLA confirmed via a sample of recent vendor additions - [ ] Trust-center feeds providing evidentiary coverage that reduces duplicate evidence collection: at least one example of a trust-center attestation used as evidence in an IR record in place of a manual screenshot or interview - [ ] Attestation-exception queue triage within 1 business day: deviations from continuous attestation triaged and either opened as findings or accepted as non-material with documented rationale - [ ] Vendor-cooperation depth improved: documented cases where trust-center integrations reduced evidence-collection burden for regulatory requests or auditor inquiries

Outcome Metrics:

Metric	Baseline	Current	Target	Met?	Notes
% Critical vendors with real-time attestation	___%	___%	≥90%	☐	Attestation telemetry
External adoption of config schemas	0	tracked	trending up	☐	External telemetry
% vendors with trust-center feed ingested	___%	___%	≥70% Critical/High	☐	Integration registry
IR reviewer-hours per integration trending down	___ hrs	___ hrs	trending down QoQ	☐	Reviewer time tracking

Metric Collection Guidance: - Trust-center feed coverage: count Critical/High vendors with a trust-center feed ingested divided by total Critical/High vendors. Source: integration registry - DPA delta auto-flag rate: count trust-center artifact changes that auto-generated a review flag divided by total detected trust-center artifact changes. Source: ingestion pipeline × IM-Vendors - Onboarding SLA compliance: % of new vendor onboardings where the trust-center feed was active within 30 days. Source: ingestion backlog × vendor onboarding dates - IR reviewer-hours: total hours logged for Critical-tier vendor IR activities per year. Source: reviewer time tracking

Evidence Location: _________

Validation Date: _________

Notes: _______

Summary Scorecard

Level	Question	Score	Weight
L1	Q1: Per-Archetype Vendor IR Checklist	___	33%
L1	Q2: Review Triggers and Timing	___	33%
L1	Q3: Findings Tracking and Closure	___	33%
L1 Total		___
L2	Q4: API-Based Config Monitoring for Critical Vendors	___	33%
L2	Q5: Automated No-Train and Retention Verification	___	33%
L2	Q6: Tier-Calibrated Vendor IR Cadence	___	33%
L2 Total		___
L3	Q7: Real-Time Configuration Attestation	___	33%
L3	Q8: Contribute Vendor Config-Baseline Schemas	___	33%
L3	Q9: Trust-Center Integration and Vendor Cooperation	___	33%
L3 Total		___
Overall IR-Vendors Score		___

Current Maturity Level: ___

Key Findings:

Priority Improvements:

Document Version: HAIAMM v3.0 Practice: Implementation Review (IR) Domain: Vendors Questionnaire Date: 2026-05-15 Author: Verifhai

Instructions:

Answer based on current practices, not plans
“Yes” requires documented evidence
Complete all Level 1 questions before Level 2
Partial implementation = “No”

↓ Download as Markdown