Strategy & Metrics (SM) - Vendors Assessment

Assessment questionnaire for measuring maturity. Answer each question honestly based on current, implemented practices.

Strategy & Metrics (SM) - Vendors Domain

HAIAMM Assessment Questionnaire v3.0

Canonical source-of-truth: ../practices/SM-Vendors-OnePager.md. This questionnaire's questions, evidence requirements, and outcome metrics are derived from that one-pager. The canonical v3.0 model: ../HAIAMM-v3.0-Framing.md.

Practice: Strategy & Metrics (SM) Domain: Vendors Purpose: Stand up a Vendor AI Assurance program that discovers, inventories, and strategically governs all AI/HAI tools and services provided by vendors, with shadow AI prevention as the primary L1 outcome and a defensible risk-tier rubric as the primary L2 deliverable. Scoring Model: Evidence + Outcome Metrics (see Scoring Methodology below)

Instructions

Answer each question honestly based on current, implemented practices (not plans or aspirations)
Each question has two components: Evidence (what you did) and Outcome Metrics (how well it worked)
Scoring uses 4 tiers: Fully Mature (1.0), Implemented (0.67), Partial (0.33), Not Implemented (0.0)
Answer progressively, Complete all Level 1 questions before Level 2
Level progression, Achieve ALL questions at lower level before advancing
Baseline first, Record current metric values before setting targets

Scoring Methodology

Tier	Score	Criteria
Fully Mature	1.0	Evidence complete + ≥3 outcome metrics meet targets
Implemented	0.67	Evidence complete + 2 outcome metrics meet targets
Partial	0.33	Evidence partially complete + <2 outcome metrics meet targets
Not Implemented	0.0	No substantive evidence of practice

Practice maturity level achieved = the highest level where all 3 questions score ≥ 0.67.

Maturity Level 1

Objective: Stand up the Vendor AI Assurance program, build an AI vendor inventory, and establish baseline metrics that prove shadow AI is decreasing

Question 1: Charter the Vendor AI Assurance program

Q1.1: Do you have a published Vendor AI Assurance program charter that names the problem (shadow AI, uncontrolled data sharing with AI vendors, unassessed third-party AI risk), defines the six in-scope AI vendor categories, names an executive sponsor (typically CISO, CIO, or Chief Procurement Officer, co-sponsored by Legal/Privacy), establishes a cross-functional working group, and defines decision rights for approval, block, and exception?

Evidence Required: - [ ] Published program charter with named executive sponsor (CISO / CIO / CPO co-sponsored by Legal/Privacy) - [ ] Problem statement covering rapid adoption, hidden AI features in existing SaaS, data-leakage by design, and regulatory exposure under EU AI Act deployer duties - [ ] Six in-scope AI vendor categories listed: consumer GenAI, AI-embedded SaaS, AI coding assistants, AI APIs/models, AI agent platforms, AI-native point solutions - [ ] Working group roster: Security, Procurement, Legal/Privacy, IT, Data Governance, one business-unit representative - [ ] Decision rights defined: who approves an AI vendor, who blocks one, who handles exceptions - [ ] Year-one success definition with a numerical target for L1 outcome metrics (e.g., "≥90% of AI vendors in use discovered and inventoried within 12 months")

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |---|---|---|---|---|---| | AI vendor inventory coverage (% of discovered AI vendors in the inventory) | ___ | ___ | ≥90% within 12 months | ☐ | | | Shadow AI ratio (unsanctioned AI vendors ÷ total AI vendors in use) | ___ | ___ | ≤15% and trending down | ☐ | | | % employees covered by an acknowledged AI Acceptable Use Policy | ___ | ___ | ≥95% of workforce | ☐ | | | Known data-exposure events to unsanctioned AI tools (per quarter) | ___ | ___ | trending down QoQ | ☐ | |

Metric Collection Guidance: - Inventory coverage: Reconcile inventory count against all discovery-source signals (expense/procurement, SSO/IdP, DNS/egress, endpoint MDM/EDR, SaaS admin consoles). Formula: inventory_count / discovered_count × 100 - Shadow AI ratio: From inventory status field, count AI vendors in use with status not "Sanctioned"; divide by total AI vendors in use across all discovery sources - AUP attestation: HR/LMS acknowledgment records for the AI Acceptable Use Policy; denominator is total workforce headcount; updated each quarter - Data-exposure events to unsanctioned AI: Aggregate DLP alerts, incident-tracker entries for AI-related data-sharing events to unsanctioned tools; count per quarter and trend

Answer: - ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets) - ☐ Implemented (Evidence complete + 2 metrics meet targets) - ☐ Partial (Evidence partially complete + <2 metrics meet targets) - ☐ Not Implemented (No evidence)

Evidence Location: _____ Metric Validation Date: ____ Notes: __________

Question 2: Build the AI vendor inventory and discover shadow AI

Q1.2: Do you maintain a single AI vendor inventory seeded from expense and procurement data, SSO/IdP sign-in logs, DNS/proxy egress logs for AI domains, MDM/EDR endpoint app inventory, and SaaS admin consoles for AI features in already-approved vendors, covering all six in-scope categories with a minimum field set including vendor name, AI capability, business owner, data classes sent, contract/DPA status, and approval status?

Evidence Required: - [ ] Single authoritative AI vendor inventory with minimum fields: vendor name, product, AI capability, business owner, user count, data classes sent (public/internal/confidential/regulated), contract/DPA status and AI-specific addendum status, vendor-trains-on-customer-data flag, risk tier assignment, approval status (Sanctioned/Provisional/Under review/Prohibited) - [ ] Expense and procurement data used as discovery signal: credit-card reports, SaaS expense platforms (Vendr, Zylo, Torii, Productiv), AP ledgers filtered for AI vendors - [ ] SSO/IdP app catalog and sign-in logs (Okta/Entra) reviewed for AI SaaS sign-in activity - [ ] DNS/proxy egress logs monitored for high-signal AI domains (openai.com, anthropic.com, gemini.google.com, cursor.sh, etc.); CASB used if available - [ ] MDM/EDR endpoint app inventory filtered for AI desktop apps and browser extensions - [ ] SaaS admin consoles audited for AI features in already-approved vendors (Notion AI, Slack AI, Zoom AI Companion, M365 Copilot, Google Workspace Gemini); amnesty window publicized through internal communications

Metric Collection Guidance: - Inventory coverage: Monthly reconciliation comparing inventory records to all discovery-source outputs; unmatched signals (new AI domain in egress, new app on MDM, new expense line) are shadow-AI candidates; report gap as coverage % - Shadow AI ratio: Filter inventory for vendors in use with approval status != "Sanctioned"; divide by total in-use vendors across all sources; trend quarterly - AUP attestation: HR/LMS query for workforce who completed AI AUP acknowledgment; denominator is total workforce headcount from HR system; updated each quarter - Data-exposure events: DLP system events for AI-tool data-sharing incidents and incident-tracker AI-related entries; count per quarter; trend downward is the goal

Evidence Location: _____ Metric Validation Date: ____ Notes: __________

Question 3: Establish foundational metrics and the shadow AI scoreboard

Q1.3: Do you baseline and report quarterly to the executive sponsor a shadow AI scoreboard covering inventory state (total/sanctioned/provisional/prohibited), new vendors discovered and their intake status, shadow AI ratio trend over the last four quarters, AUP attestation coverage, and the top five data-exposure risks to unsanctioned AI tools with remediation owners?

Evidence Required: - [ ] Quarterly shadow AI scoreboard published and delivered to the executive sponsor, at least two consecutive quarters on record - [ ] Scoreboard covers: AI vendors in inventory (total/sanctioned/provisional/prohibited), new AI vendors discovered this quarter and intake status, shadow AI ratio trend (last 4 quarters), AUP attestation coverage, top 5 data-exposure risks to unsanctioned AI tools with remediation owner - [ ] Shadow AI ratio trended over last 4 quarters with commentary on direction - [ ] AUP attestation percentage reported with workforce headcount denominator - [ ] Intake SLA tracked: new AI vendor intake triaged within 5 business days; inventory freshness at ≥80% of records reviewed/updated in the last 90 days

Metric Collection Guidance: - Scoreboard delivery cadence: Confirm last two quarters have a dated scoreboard delivered to exec sponsor with their acknowledgment on record - Shadow AI ratio trend: Four-quarter chart or table; downward trend is the L1 success signal; source is inventory status field reconciled monthly against discovery sweeps - AUP attestation: Percentage with workforce headcount denominator explicitly stated; HR/LMS is the authoritative source; updated each quarter - Top-5 data-exposure risks: Each entry lists the unsanctioned vendor or data-exposure scenario, named remediation owner, and current status (open / in-progress / mitigated) - Intake SLA: Intake tracker report showing % of new vendor requests triaged within 5 BD; any backlog older than 5 BD flagged

Evidence Location: _____ Metric Validation Date: ____ Notes: __________

Maturity Level 2

Objective: Risk-tier the AI vendor inventory, calibrate the program's intensity per tier, and measure practice maturity and shadow-AI reduction per tier, not only in aggregate

Question 1: Define the AI vendor risk-tier rubric

Q2.1: Do you have a published risk-tier rubric (Critical / High / Medium / Low) assigning a tier to every AI vendor in the inventory based on six auditable dimensions, data sensitivity reaching the vendor, decision-affecting use (GDPR Art. 22 / EU AI Act Annex III), agentic capability, user exposure, regulatory scope, and concentration/criticality, with tier derivation deterministic from inputs, human overrides recorded with rationale, and 100% of inventory records carrying a current tier?

Evidence Required: - [ ] Published tier-rubric document listing all six auditable dimensions with deterministic assignment logic (not reviewer vibes) - [ ] 100% of inventory records carry a current tier assignment derived from the rubric - [ ] Data sensitivity dimension: regulated data (PHI/PCI/regulated PII/source code/customer confidential) reaching the vendor → Critical or High - [ ] Decision-affecting use dimension: EU AI Act Annex III high-risk use or GDPR Art. 22 materially affected decision → Critical - [ ] Agentic capability dimension: agent platform or multi-tool function surface acting on org systems → elevate tier - [ ] Regulatory scope dimension: sector-specific triggers (HIPAA BAA, PCI service provider, FINRA/SEC model risk, HHS/FDA clinical) → elevate - [ ] Human override log maintained: all overrides recorded with rationale and reviewed by working group

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |---|---|---|---|---|---| | % of inventory with a current tier assignment | ___ | ___ | 100% | ☐ | | | Tier-treatment matrix adherence, % Critical vendors with full-scope treatment completed in last 12 months | ___ | ___ | ≥95% | ☐ | | | Tier-weighted shadow AI ratio (Critical-weighted) | ___ | ___ | Critical = 0 unsanctioned; overall trending down | ☐ | | | Per-tier SLA adherence (intake, DR, IR, ST, ML) | ___ | ___ | ≥90% per tier | ☐ | | | Tier drift rate (tier changes per year) | ___ | ___ | tracked; unexplained changes = 0 | ☐ | |

Metric Collection Guidance: - % inventory with tier assignment: Automated check, records with null risk_tier flagged for remediation within 5 BD; stale tier (not re-confirmed after a material change) also flagged - Tier-treatment matrix adherence: Cross-reference Critical-tier vendor records against evidence of full-scope treatment: full SR pack completed, per-vendor TA deep model completed, DR completed, IR within cadence, full ST battery complete - Tier-weighted shadow AI ratio: Critical-tier vendors in use with status != "Sanctioned" must be 0; overall ratio across all tiers should trend down; both reported in the quarterly scoreboard - Per-tier SLA adherence: From intake tracker, DR queue, IR schedule, ST tracker, and ML system; % of SLA-bound actions completed on time per tier; report monthly - Tier drift rate: Governance log reviewed at each working-group meeting; any change without a recorded dimension-change rationale is unexplained; target = 0

Evidence Location: _____ Metric Validation Date: ____ Notes: __________

Question 2: Calibrate program intensity per tier

Q2.2: Do you have a published tier-treatment matrix defining differential program intensity across all downstream Vendors-domain practices (PC, TA, DR, IR, ST, ML) for each tier, and is this matrix enforced, with Critical-tier vendors receiving full SR pack + full REM at intake, a per-vendor deep TA model, full-lane DR, semi-annual IR plus review on material change, a full ST battery with quarterly red-team probe, all detections tuned, and mandatory re-review within 14 days of any vendor material change?

Evidence Required: - [ ] Tier-treatment matrix published covering all downstream practices with explicit treatment per tier - [ ] Critical-tier treatment documented: full SR pack + full REM; per-vendor deep TA model; full-lane DR required; semi-annual IR + on change; full ST battery + quarterly red-team probe; all detections tuned for the integration; mandatory re-review within 14 days of material change - [ ] Low-tier fast-track documented: base SR pack only; archetype TA snapshot; no DR required; go-live IR only; spot-check ST; baseline logging; re-review at annual review only - [ ] Evidence that Critical-tier vendors are actually receiving full-scope treatment (vendor × treatment evidence in program tracker for ≥95% of Critical vendors in last 12 months) - [ ] Downstream practices (PC, TA, DR, IR, ST, ML) each acknowledged the calibration via working-group decision record

Metric Collection Guidance: - Tier-treatment matrix adherence: Build a cross-reference table, for each Critical-tier vendor, list last DR date, last IR date, ST coverage status, ML detection status; ≥95% must show all Critical-tier treatments completed in last 12 months - Per-tier SLA adherence: Aggregate SLA data from intake, DR, IR, and ST trackers; calculate on-time completion % per tier; report monthly - Tier-weighted shadow AI ratio: Critical-tier unsanctioned vendors must be 0; overall shadow AI ratio should decrease; both reported in quarterly scoreboard - Tier drift rate: Governance log cross-checked at each working-group meeting; unexplained tier changes (no dimension-change rationale) flagged for sponsor visibility; target = 0

Evidence Location: _____ Metric Validation Date: ____ Notes: __________

Question 3: Per-tier scoreboard and governance

Q2.3: Does the quarterly shadow AI scoreboard report inventory state per tier with Critical-tier unsanctioned AI explicitly tracked at zero, include a tier-movement log with rationale, report per-tier SLA adherence for intake/DR/IR/ST/ML, and is it reviewed by the executive sponsor who explicitly discusses tier-balance?

Evidence Required: - [ ] Quarterly scoreboard includes a tier-level breakdown showing vendors by tier (Critical/High/Medium/Low) with sanctioned/provisional/prohibited counts per tier - [ ] Critical-tier unsanctioned AI vendors in use is a named metric in the scoreboard; target is 0; any non-zero value is a headline finding requiring sponsor action - [ ] Tier-movement log included: vendors that moved up or down in the quarter, with dimension(s) that changed and rationale for each move - [ ] SLA adherence per tier reported for intake, DR, IR, ST, and ML - [ ] Quarterly executive review documented (agenda + minutes) showing tier-balance discussion: is the program's effort matching the risk profile? - [ ] Tier calibration exercise documented: at least quarterly, a sample of 20 vendors re-tiered by a second reviewer; drift tracked

Metric Collection Guidance: - Per-tier scoreboard delivery: Last two consecutive quarterly scoreboards must include a tier-level breakdown (not just aggregate); each scoreboard shows delta from prior quarter - Critical-tier unsanctioned count: Named metric in each scoreboard; source is inventory filtered on tier=Critical AND status != Sanctioned AND in-use; target = 0 - Tier-movement log completeness: Each entry must have vendor name, prior tier, new tier, dimension(s) that changed, reviewer name, and date; unexplained changes target = 0 - SLA adherence per tier: Pulled from intake, DR, IR, ST, and ML systems; aggregated per tier; reported as % on-time per tier per quarter - Executive review: Filed governance document confirming exec sponsor reviewed tier-balance section and issued follow-up actions or signed off

Evidence Location: _____ Metric Validation Date: ____ Notes: __________

Maturity Level 3

Objective: Automate inventory and tier maintenance from live signals, benchmark the program against external peers, and contribute anonymized AI-vendor ecosystem intelligence back to the industry

Question 1: Continuous inventory and tier automation

Q3.1: Does the AI vendor inventory auto-update from live expense/procurement feeds, SSO/IdP app catalog, DNS/egress telemetry, SaaS admin audit feeds, endpoint AI-tool inventory, and intake, with tier assignments rule-based on versioned, replayable logic, tier changes adjusting automatically when dimensional inputs change, and a published data-quality SLO of ≥99% of active AI vendors correctly tiered within 48 hours of a material change?

Evidence Required: - [ ] Published data-quality SLO: ≥99% of active AI vendors correctly tiered within 48 hours of a material change - [ ] Automated feeds operational: expense/procurement feeds, SSO/IdP app catalog, DNS/egress telemetry, SaaS admin audit feeds (for parent vendors like M365, Slack, Notion), endpoint AI-tool inventory from MDM/EDR, intake system, self-attestation - [ ] Tier rules documented as versioned, replayable logic; rule changes are change-logged and can be replayed against historical inventory state - [ ] Human curation queue defined for: new archetypes, ambiguous discoveries, dimensional-input conflicts - [ ] Automation health dashboard: signal-feed freshness and error rate monitored; on-call paged when a feed staleness threshold is exceeded - [ ] Tier changes adjust automatically when dimensional inputs change (e.g., a vendor gains agentic capability, data sensitivity classification changes); humans intervene only on exception cases

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |---|---|---|---|---|---| | Inventory auto-update latency | ___ | ___ | ≤48h for material changes | ☐ | | | % inventory entries auto-curated vs. human-curated | ___ | ___ | ≥80% auto | ☐ | | | External benchmarks tracked | ___ | ___ | ≥5 | ☐ | | | Industry contributions per year | ___ | ___ | ≥4 substantive | ☐ | | | Executive-level ROI narrative refreshed with external benchmarks | ___ | ___ | semi-annual | ☐ | |

Metric Collection Guidance: - Auto-update latency: Measure time from a known material change event (new AI vendor approved in SSO, new AI domain appears in egress logs, new expense line for an AI vendor) to the corresponding inventory record update; P95 across sampled events per quarter - % auto-curated: From the inventory curation log, count records updated by automated feeds vs. human-initiated edits; report as a ratio per quarter - Automation health: Signal-feed freshness report showing last successful pull time for each feed; alert thresholds defined and on-call paging tested - Data-quality SLO adherence: At each quarter-end, spot-check 20 vendors against their expected tier from the rubric; report % correctly tiered

Evidence Location: _____ Metric Validation Date: ____ Notes: __________

Question 2: External benchmarking

Q3.2: Do you publish a semi-annual external-benchmarking brief comparing the program against at least five peer-comparable metrics via sector ISACs with AI-vendor working groups, ISO/IEC 42001 community, NIST AI RMF implementations, CSA AI Safety Initiative, Shared Assessments AI-vendor track, and formal peer roundtables, and do benchmark deltas explicitly inform program investment decisions?

Evidence Required: - [ ] Semi-annual benchmarking brief published, two most recent on file with dates, each containing ≥5 peer-comparable metrics from named external sources - [ ] Benchmarking sources include at least two of: sector ISACs (FS-ISAC, H-ISAC, IT-ISAC) AI-vendor working groups / ISO/IEC 42001 community / NIST AI RMF implementations / CSA AI Safety Initiative / Shared Assessments AI-vendor track / formal peer roundtables - [ ] Metrics benchmarked cover: shadow AI ratio, AUP coverage, intake SLA, per-tier depth of treatment, automation level, and procurement cycle time for approved AI vendors - [ ] Benchmark deltas explicitly referenced in a program investment or prioritization decision; documentation filed within 90 days of each brief - [ ] "How we compare" brief refreshed semi-annually; peer selection rationale documented, peers chosen to stretch, not flatter

Metric Collection Guidance: - External benchmarks tracked: Each brief lists ≥5 named benchmark data points; each traceable to an ISAC report, ISO/IEC 42001 community publication, NIST AI RMF guidance, CSA controls document, or named peer roundtable - Benchmark-driven investment: Program planning or budget document explicitly citing a benchmark delta as rationale; filed within 90 days of each brief - Semi-annual cadence: Two briefs within a 12-month window; no gap > 7 months between consecutive briefs - Executive ROI narrative: Annual exec/board briefing deck or memo includes benchmark comparisons and avoided-loss examples; procurement cycle time for approved AI vendors is a key effectiveness metric

Evidence Location: _____ Metric Validation Date: ____ Notes: __________

Question 3: Contribute anonymized AI-vendor ecosystem intelligence

Q3.3: Does the program contribute at least four substantive, anonymized artifacts per year to the AI-vendor assurance ecosystem through MITRE ATLAS, OWASP LLM / Agentic Top 10, NIST AI RMF Playbook, AI Vulnerability Database submissions, or sector ISAC feeds, with each contribution anonymized, legally vetted, and traceable to a published working-group output or advisory?

Evidence Required: - [ ] Contribution log maintained listing all submissions: target body (MITRE ATLAS, OWASP, NIST, AVID, ISAC), submission type (new AI-vendor-ecosystem TTP, real-world telemetry, discovery signal pattern, detection pattern), date submitted, anonymization review completed, status - [ ] At least 4 substantive contributions per year in the most recent 12-month window; each is a technical artifact accepted or in active review by the named body; conference talks and press releases do not count - [ ] Each contribution has a legal/privacy review sign-off confirming anonymization before submission - [ ] Contributions traceable to published outputs: MITRE ATLAS technique entries, OWASP review comments incorporated, NIST AI RMF Playbook references, AVID entries, ISAC advisory feeds - [ ] Contribution pipeline shows ≥2 items in-flight (draft, in-review, or being prepared) at any working-group review

Metric Collection Guidance: - Industry contributions per year: Count entries in the contribution log for trailing 12 months where status = submitted or accepted to a named body; only substantive technical artifacts count - Contribution pipeline health: At any working-group meeting, pipeline log shows ≥2 items not yet in submitted status; noted in working-group minutes - Legal/privacy review: Each contribution log entry must have reviewer name and date; no contribution submitted without this sign-off - Executive ROI narrative: Filed annually to exec/board; references external benchmarks and avoided-loss examples (data-exposure incidents avoided, regulatory exposure mitigated, procurement cycle time reduced); sponsor decisions citing benchmark data indicate program maturity

Evidence Location: _____ Metric Validation Date: ____ Notes: __________

Summary Scorecard

Level	Q1	Q2	Q3	Avg	Achieved?
L1	__	__	__	__	☐
L2	__	__	__	__	☐
L3	__	__	__	__	☐

Practice maturity level achieved: ___ (highest level where all 3 questions score ≥ 0.67)

Document Version: HAIAMM v3.0 Practice: Strategy & Metrics (SM) Domain: Vendors Last Updated: 2026-05-15 Author: Verifhai

Instructions:

Answer based on current practices, not plans
“Yes” requires documented evidence
Complete all Level 1 questions before Level 2
Partial implementation = “No”

↓ Download as Markdown