Strategy & Metrics (SM)

Vendors Domain - HAIAMM v3.0

Practice Overview

Objective: Stand up a Vendor AI Assurance program that discovers, inventories, and strategically governs all AI/HAI tools and services provided by vendors, with shadow AI prevention as the primary L1 outcome.

Description: The Vendors domain governs AI capabilities the organization consumes from third parties: consumer GenAI (ChatGPT, Claude, Gemini, Copilot), AI features silently enabled inside existing SaaS (Notion AI, Slack AI, Zoom AI Companion, M365 Copilot), AI coding assistants (Cursor, Windsurf, Tabnine), AI APIs, downloaded open-source models, and AI agent platforms. SM establishes the program charter, the AI vendor inventory, and the practice-maturity metrics that prove the program is working.

Context: Employees and teams adopt AI vendors faster than procurement, legal, and security can review them. Free tiers, credit-card SaaS, and "turn on AI" toggles inside already-approved vendors create a shadow AI footprint that bypasses data-handling, contractual, and regulatory controls (EU AI Act, GDPR Art. 28, SOC 2 CC9.2, ISO/IEC 42001). The Vendor AI Assurance program makes this footprint visible and puts a light-touch intake on the path to adoption, so sanctioned AI can accelerate and unsanctioned AI cannot quietly accumulate risk.

Maturity Level 1

Objective: Stand up the Vendor AI Assurance program, build an AI vendor inventory, and establish baseline metrics that prove shadow AI is decreasing

At this level, the organization makes AI vendor use visible, assigns accountability, and begins measuring the reduction of shadow AI across the employee population.

Dependencies

• None, entry-point practice for the Vendors domain. SM-Vendors L1 precedes all other Vendors-domain L1s.
• Alignment (not a hard dependency): enterprise-wide SM strategy and CISO governance structure, so the AI vendor program plugs into existing risk committees rather than forming a parallel stack.
• Supports / unblocks: PC-Vendors L1 (policies need the inventory), TA-Vendors L1 (threat modeling needs the inventory), IM-Vendors L1 (incident routing needs the owner + sponsor structure), ML-Vendors L1 (monitoring needs the inventory).

Desired Outcomes

• Shadow AI footprint is visible, attributable to business owners, and trending down quarter-over-quarter.
• A single AI vendor inventory is the authoritative source of truth across Security, Procurement, Legal/Privacy, and IT.
• An accountable executive owns AI vendor risk; decision rights for approval, block, and exception are unambiguous.
• Practice maturity is measurable from a small, automatable metric set rather than from activity counts.
• The program is positioned as an enabler (fast-track for pre-approved vendors), not a bottleneck, so business units work through it rather than around it.

Activities

A) Charter the Vendor AI Assurance program

Publish a short program charter that names the problem (shadow AI, uncontrolled data sharing with AI vendors, unassessed third-party AI risk), defines scope, and assigns accountable ownership. The program does not need a new team, it needs a named owner and a small cross-functional working group.

Charter elements: - Problem statement, why AI vendors are a distinct third-party risk category (rapid adoption, hidden AI features, data-leakage by design, regulatory exposure under EU AI Act deployer duties) - In-scope AI vendor categories, consumer GenAI, AI-embedded SaaS, AI coding assistants, AI APIs/models, AI agent platforms, AI-native point solutions - Executive sponsor, typically CISO, CIO, or Chief Procurement Officer; co-sponsored by Legal/Privacy - Working group, Security, Procurement, Legal/Privacy, IT, Data Governance, one business-unit representative - Decision rights, who can approve an AI vendor, who can block one, who handles exceptions - Success definition for year one, a numerical target for the L1 outcome metrics below (e.g., "90% of AI vendors in use are discovered and inventoried within 12 months")

B) Build the AI vendor inventory and discover shadow AI

Establish a single AI vendor inventory as the program's source of truth. Seed it from authoritative sources, then actively discover shadow AI using signals already available to IT and Security, no new tooling required at L1.

Inventory fields (minimum): - Vendor name, product, AI capability in scope (chat, code gen, image, agent, embedded-feature) - Business owner and user count - Data classes sent to the vendor (public, internal, confidential, regulated) - Contract/DPA status and AI-specific addendum status - Whether the vendor trains on customer data (default on/off, opt-out available) - Risk tier assignment (see Activity C) - Approval status: Sanctioned / Provisional / Under review / Prohibited

Discovery sources (at L1, use what you already have): - Expense and procurement data, credit-card reports, SaaS expense platforms (Vendr, Zylo, Torii, Productiv), AP ledgers, grep for AI vendors - Identity and SSO, Okta/Entra app catalog and sign-in logs for AI SaaS - Network/egress, DNS/proxy logs for high-signal AI domains (openai.com, anthropic.com, gemini.google.com, cursor.sh, etc.); CASB if available - Endpoint, MDM/EDR app inventory for AI desktop apps and browser extensions - Existing SaaS admin consoles, enable/audit AI features in already-approved vendors (Notion AI, Slack AI, Zoom AI Companion, M365 Copilot, Google Workspace Gemini) - Self-attestation, a 60-second intake form publicized through internal comms; amnesty window for disclosing unsanctioned use

C) Establish foundational metrics that measure practice maturity and shadow AI reduction

Baseline and track a small set of outcome, process, and effectiveness metrics. Keep L1 metrics simple, automatable, and directly tied to the L1 outcome (shadow AI reduction).

Outcome metrics (lagging, measure the goal): | Metric | Baseline | L1 Target | Source | |---|---|---|---| | AI vendor inventory coverage (% of discovered AI vendors in the inventory) | measure | ≥90% within 12 months | Inventory vs. discovery-source reconciliation | | Shadow AI ratio (unsanctioned AI vendors ÷ total AI vendors in use) | measure | ≤15% and trending down | Inventory status field | | % employees covered by an acknowledged AI Acceptable Use Policy | measure | ≥95% of workforce | HR / LMS attestation | | Known data-exposure events to unsanctioned AI tools (per quarter) | measure | trending down quarter-over-quarter | DLP, incident tracker |

Process metrics (leading, predict whether outcomes land): - Discovery cadence, shadow AI discovery sweeps run at least monthly - Intake SLA, new AI vendor intake triaged within 5 business days - Inventory freshness, ≥80% of inventory records reviewed/updated in the last 90 days

Effectiveness metrics (business value): - Procurement cycle time for approved AI vendors (should decrease as the program matures, the program is not a bottleneck) - % of AI vendor approvals reused across business units (reuse indicates the program scales) - Number of avoided shadow-AI incidents tied to early discovery (documented cases)

Shadow AI scoreboard (published quarterly to the exec sponsor): 1. AI vendors in inventory (total / sanctioned / provisional / prohibited) 2. New AI vendors discovered this quarter and their intake status 3. Shadow AI ratio trend (last 4 quarters) 4. AUP attestation coverage 5. Top 5 data-exposure risks to unsanctioned AI tools, with remediation owner

Success criteria: - Program charter published and sponsored by an accountable executive - AI vendor inventory exists as a single source of truth with ≥90% coverage of discovered AI vendors within 12 months - Shadow AI ratio baselined and trending down for two consecutive quarters - ≥95% of employees have acknowledged an AI Acceptable Use Policy - Quarterly shadow AI scoreboard delivered to the executive sponsor

Maturity Level 2

Objective: Risk-tier the AI vendor inventory, calibrate the program's intensity per tier, and measure practice maturity and shadow-AI reduction per tier, not only in aggregate

At this level, the Vendor AI Assurance program stops treating every AI vendor the same. Risk tiers drive how deep intake goes, how often reviews happen, which detections fire, and what the sponsor sees on the scoreboard. Shadow AI is measured and actively reduced against the tier-weighted profile, because one uncontrolled Critical-tier AI vendor is not equivalent to five Low-tier ones.

Dependencies

• SM-Vendors L1 (required): inventory, charter, and baseline metrics are the substrate L2 tiers and calibrates.
• PC-Vendors L1 (required): priority compliance map drives one of the tier dimensions (regulated / high-risk AI Act / GDPR Art. 22 use cases).
• TA-Vendors L1 (required): threats drive another tier dimension (agentic capability, data exposure, output-misuse surface).
• Supports / unblocks: PC-Vendors L2 (tier-driven policy depth), TA-Vendors L2 (per-tier deep threat modeling), DR-Vendors L2, IR-Vendors L2, ST-Vendors L2, ML-Vendors L2 (all per-tier calibrated).

Desired Outcomes

• Every AI vendor in the inventory carries a risk-tier assignment tied to explicit, auditable dimensions, not reviewer vibes.
• Program intensity is visibly differentiated: Critical gets the full program, Low gets the fast-track, and no one confuses the two.
• The quarterly shadow AI scoreboard splits by tier; the sponsor can see which tiers are healthy and which are drifting.
• Tier movements (upgrades or downgrades) are a normal, tracked event, not hidden reclassifications.
• Practice maturity is now defensible per tier: "we are mature at Critical and still building at Medium" is a real, evidenced statement.

Activities

A) Define the AI vendor risk-tier rubric

Four tiers, Critical / High / Medium / Low, assigned from a small set of auditable dimensions: - Data sensitivity reaching the vendor, regulated (PHI / PCI / regulated PII / source code / customer confidential) → Critical or High. - Decision-affecting use, AI output materially drives a decision with legal or significant effect on a person (GDPR Art. 22) or a high-risk use case under EU AI Act Annex III → Critical. - Agentic capability, the vendor includes an agent platform or multi-tool function surface acting on org systems → elevate tier. - User exposure, customer-facing use → elevate; internal-only employee-facing use → neutral. - Regulatory scope, sector-specific (HIPAA BAA applicable, PCI service-provider, FINRA/SEC model risk, HHS/FDA clinical) → elevate. - Concentration / criticality, AI vendor is load-bearing for a customer-facing product or an internal critical workflow → elevate.

Rubric is documented as a short table; tier is derived deterministically from the inputs; human overrides are allowed but recorded with rationale.

B) Calibrate program intensity per tier

Publish a tier-treatment matrix, what each tier gets from the program. Example calibration:

Each downstream Vendors-domain L2 practice inherits this calibration and defines its own tier-specific activities.

C) Per-tier scoreboard and governance

The L1 shadow AI scoreboard becomes tier-aware at L2: - Shadow AI ratio reported by tier (a Critical-tier unsanctioned AI vendor is its own headline; a Low-tier one is a line item). - SLA adherence per tier (intake, DR, IR, ST, ML) reported monthly. - Tier-movement log, vendors that moved up a tier in the quarter (tighter treatment now applies) and those that moved down (with rationale). - Quarterly executive review explicitly discusses tier-balance, is the program's effort matching the program's risk profile?

Outcome Metrics (L2)

Process Metrics (leading)

• Tier-rubric review cadence, reviewed every 2 quarters; changes change-logged.
• Tier calibration exercise, at least quarterly, sample of 20 vendors re-tiered by a second reviewer; drift tracked.
• Per-tier queue depth monitored; no tier's backlog exceeds a published threshold.

Effectiveness Metrics (business value)

• Effort allocation match, % of reviewer hours spent on Critical+High tiers vs. Medium+Low; should rise relative to L1.
• Avoided-incident stories where tier-differentiation caught risk earlier.
• Scoreboard drives budget, tier-level dashboards referenced in quarterly planning.

Success Criteria

• Risk-tier rubric published; tier assigned to 100% of inventory.
• Tier-treatment matrix published; downstream practices calibrated to it.
• Per-tier shadow AI ratio reported quarterly; Critical-tier unsanctioned AI count = 0.
• Per-tier SLA adherence ≥90% across practices.
• Tier-movement governance active, changes logged with rationale and reviewed by the sponsor.

Maturity Level 3

Objective: Automate inventory and tier maintenance from live signals, benchmark the program against external peers, and contribute anonymized AI-vendor ecosystem intelligence back to the industry

At this level, the program is predominantly signal-driven rather than ticket-driven. Inventory and tiering update from authoritative feeds; human review is exception-based. The program can state where it stands against peers on specific metrics, and it gives back to the AI-vendor assurance ecosystem through standards bodies, ISACs, and public taxonomy contributions.

Dependencies

• SM-Vendors L2 (required): tiering and calibration must be settled before automation is trustworthy.
• ML-Vendors L2+ (required): signals (egress, SSO, admin-audit, expense, endpoint) need the monitoring pipeline behind them.
• Supports / unblocks: the other 11 Vendors practices can move to L3 automation patterns because SM now supplies automated inventory and tier data.

Desired Outcomes

• Inventory accuracy is measured in hours-of-latency, not months.
• Tier assignments adjust automatically when dimensional inputs change; humans intervene only on exceptions.
• External benchmarking is routine, the program sponsor can answer "how do we compare?" with specific deltas, not hand-waving.
• The organization is a net contributor to the AI-vendor assurance ecosystem, industry references point back to us, not only outward from us.
• The program's strategic ROI is demonstrable: dollars of program investment mapped to dollars of avoided loss, with external benchmarks reinforcing the narrative.

Activities

A) Continuous inventory and tier automation

• Inventory auto-updates from: expense/procurement feeds, SSO/IdP app catalog, DNS/egress telemetry, SaaS admin audit feeds (parent vendors), endpoint AI-tool inventory, intake system, self-attestation.
• Tier assignments are rule-based on the L2 rubric inputs; rule changes are versioned and replayable.
• Human curation handles: new archetypes, ambiguous discoveries, dimensional-input conflicts.
• Data-quality SLO is published (e.g., ≥99% of active AI vendors correctly tiered within 48 hours of a material change).

B) External benchmarking

• Program metrics compared against peer benchmarks via:
• Sector ISACs that have AI-vendor working groups (FS-ISAC, H-ISAC, IT-ISAC);
• AI-vendor standards bodies (ISO/IEC 42001 community, NIST AI RMF implementations, CSA AI Safety Initiative);
• Shared Assessments AI-vendor track (as it matures);
• Formal peer roundtables.
• A published "how we compare" brief refreshed semi-annually covers: shadow AI ratio, AUP coverage, intake SLA, per-tier depth, automation level.
• Benchmark deltas inform program investment and board-level narrative.

C) Contribute anonymized AI-vendor ecosystem intelligence

• Contribute to:
• MITRE ATLAS (new AI-vendor-ecosystem TTPs);
• OWASP LLM / Agentic Top 10 (review, comment, real-world telemetry);
• NIST AI RMF Playbook and successor editions;
• AI Vulnerability Database submissions for vendor-product issues (responsibly disclosed);
• ISAC feeds on emerging AI vendors, archetypes, discovery signals, detection patterns.
• Target: minimum 4 substantive contributions per year; quality over volume; every contribution anonymized and legally vetted.

Outcome Metrics (L3)

Process Metrics (leading)

• Automation health, signal-feed freshness and error rate monitored.
• Benchmarking cadence honored (semi-annual brief).
• Contribution pipeline always has ≥2 items in-flight.

Effectiveness Metrics (business value)

• Sponsor decisions (budget, headcount, scope) citing benchmark data.
• Industry recognition, invitations to working groups, citations of the program's contributions, peer adoption of published patterns.
• Talent, the program attracts experienced reviewers because of its external profile.

Success Criteria

• Inventory auto-update SLO published and met.
• Tier-assignment automation operational with published rules and human-exception handling.
• Semi-annual external-benchmarking brief published to the sponsor.
• ≥4 substantive industry contributions per year, anonymized and cited.
• ROI narrative including external benchmarks delivered to exec/board at least annually.

Key Success Indicators

Level 1: - Vendor AI Assurance program charter published and sponsored by an accountable executive (CISO / CIO / CPO), with a cross-functional working group (Security, Procurement, Legal/Privacy, IT, Data Governance, business rep) - AI vendor inventory exists as a single source of truth, covering all in-scope categories (consumer GenAI, AI-embedded SaaS, AI coding assistants, AI APIs/models, AI agent platforms) - Shadow AI actively discovered each month from expense/SSO/egress/endpoint/SaaS-admin signals, reconciled against the inventory - AI Acceptable Use Policy acknowledged by ≥95% of the workforce - Foundational metrics baselined: inventory coverage, shadow AI ratio, AUP attestation, intake SLA; quarterly shadow AI scoreboard delivered to the exec sponsor

Level 2: - Risk-tier rubric published and applied, 100% of inventory carries a current tier. - Tier-treatment matrix published; downstream practices (PC, TA, DR, IR, ST, ML) calibrated to it. - Quarterly shadow AI scoreboard reports per tier; Critical-tier unsanctioned AI = 0. - Per-tier SLA adherence ≥90% across program activities. - Tier-movement governance operating with logged rationale.

Level 3: - Inventory auto-update latency ≤48 hours for material changes; ≥80% of curation is automated. - Tier-assignment automation operates on a published rule set with exception-based human review. - Semi-annual external-benchmarking brief published to the sponsor, citing at least five peer-comparable metrics. - ≥4 substantive anonymized industry contributions per year (MITRE ATLAS / OWASP / NIST AI RMF / AI Vulnerability Database / ISACs). - Executive / board ROI narrative refreshed at least annually with external benchmarks and documented avoided-loss examples.

Common Pitfalls

Level 1: - ❌ Inventory is seeded only from procurement records, misses free-tier, credit-card, and personal-account AI use (the bulk of shadow AI) - ❌ Treating "AI-embedded features" in existing SaaS as out of scope (Notion AI, Slack AI, Zoom AI Companion, M365 Copilot) because the underlying vendor is already approved - ❌ Program positioned as a blocker, intake SLA unpublished, procurement cycle time balloons, business units route around the program - ❌ Executive sponsor is security-only; Procurement, Legal/Privacy, and a business rep are not co-owners, so policy lacks authority - ❌ Metrics count activity (tickets processed, reviews completed) instead of outcome (shadow AI ratio down, AUP coverage up, data-exposure events down) - ❌ No amnesty window for self-disclosure, so employees hide shadow AI rather than surface it

Level 2: - ❌ Tier-rubric inputs are subjective ("important vendor," "sensitive data"), reviewers tier differently; auditors don't trust it. - ❌ Tier-treatment matrix published but not enforced, Critical vendors routed to the same queue as Low; calibration exists on paper only. - ❌ Scoreboard still reported in aggregate, hiding that Critical-tier shadow AI is present because overall averages look fine. - ❌ Tier upgrades get resistance because they trigger more work, no governance on tier-movement leaves the program stuck at initial assignments. - ❌ Downstream practices treat tier as advisory, not operational, DR/IR/ST/ML don't differentiate their scope by tier.

Level 3: - ❌ Automation runs without data-quality SLO, signal-driven inventory silently drifts and humans stop trusting it. - ❌ Benchmarking chooses peers that flatter the program instead of stretching it. - ❌ Industry "contributions" are press releases and conference talks, not technical artifacts that actually land in MITRE/OWASP/NIST/AIVD. - ❌ Automated tiering rules encode historical bias (weighted against categories the program under-reviewed, under-weighted for ones the program prefers), audit of rule inputs never happens. - ❌ ROI narrative decouples from reality, external benchmarks cited but program's own metrics are stale.

Practice Maturity Questions

Level 1: 1. Is there a published Vendor AI Assurance program charter with a named executive sponsor and a cross-functional working group (Security, Procurement, Legal/Privacy, IT, Data Governance, business rep)? 2. Does a single AI vendor inventory exist, seeded from expense/SSO/egress/endpoint/SaaS-admin signals, covering consumer GenAI, AI-embedded SaaS, AI coding assistants, AI APIs/models, and AI agent platforms, with ≥90% coverage of discovered AI vendors? 3. Are the L1 outcome metrics baselined and reported quarterly to the sponsor, AI vendor inventory coverage, shadow AI ratio, AUP attestation (≥95%), and known data-exposure events to unsanctioned AI tools?

Level 2: 1. Is every AI vendor in the inventory assigned a risk tier based on an auditable rubric covering data sensitivity, decision-affecting use, agentic capability, user exposure, regulatory scope, and concentration? 2. Is there a published tier-treatment matrix driving differential intensity across PC, TA, DR, IR, ST, and ML, with ≥95% of Critical-tier vendors receiving full-scope treatment in the last 12 months? 3. Does the quarterly shadow AI scoreboard report per tier (with Critical-tier unsanctioned AI explicitly tracked at zero), and does tier-movement get logged and reviewed by the program sponsor?

Level 3: 1. Does inventory and tier assignment auto-update from live signals (expense, SSO, egress, SaaS admin, endpoint, intake, self-attestation) with a published data-quality SLO, and is ≥80% of curation handled automatically with exception-based human review? 2. Do you publish a semi-annual external-benchmarking brief comparing the program against at least five peer-comparable metrics via ISACs, standards bodies, or industry roundtables, and does it drive program investment decisions? 3. Does the program contribute at least four substantive, anonymized artifacts per year to the AI-vendor assurance ecosystem (MITRE ATLAS, OWASP LLM/Agentic, NIST AI RMF, AI Vulnerability Database, sector ISACs), and does the exec/board ROI narrative cite external benchmarks?

Document Version: HAIAMM v3.0 Practice: Strategy & Metrics (SM) Domain: Vendors Last Updated: 2026-05-12 Author: Verifhai