Assessment questionnaire for measuring maturity. Answer each question honestly based on current, implemented practices.
Canonical source-of-truth:
../practices/SM-Software-OnePager.md. This questionnaire's questions, evidence requirements, and outcome metrics are derived from that one-pager. The canonical v3.0 model:../HAIAMM-v3.0-Framing.md.
Practice: Strategy & Metrics (SM) Domain: Software Purpose: Stand up an AI/HAI Software Assurance program that discovers, inventories, and strategically governs all AI/HAI software the organization builds, with shadow-AI-in-engineering prevention as the primary L1 outcome and a defensible risk-tier rubric as the primary L2 deliverable. Scoring Model: Evidence + Outcome Metrics (see Scoring Methodology below)
| Tier | Score | Criteria |
|---|---|---|
| Fully Mature | 1.0 | Evidence complete + ≥3 outcome metrics meet targets |
| Implemented | 0.67 | Evidence complete + 2 outcome metrics meet targets |
| Partial | 0.33 | Evidence partially complete + <2 outcome metrics meet targets |
| Not Implemented | 0.0 | No substantive evidence of practice |
Practice maturity level achieved = the highest level where all 3 questions score ≥ 0.67.
Objective: Stand up the AI/HAI Software Assurance program, build an inventory of AI/HAI software the organization builds, and establish baseline metrics that prove shadow AI in engineering is decreasing
Q1.1: Do you have a published AI/HAI Software Assurance program charter that names the problem (shadow AI in engineering, ungoverned LLM integrations, agents shipped without threat modeling), defines the seven in-scope AI/HAI software archetypes, names an executive sponsor (CISO + CTO / Head of Engineering / Chief AI Officer), establishes a cross-functional working group, and defines decision rights for approval, block, exception, and go-live?
Evidence Required: - [ ] Published program charter document with named executive sponsor (CISO + CTO or equivalent) and co-signature from Privacy/Legal where applicable - [ ] Problem statement explicitly covering AI-specific failure modes (prompt injection, training-data leakage, tool misuse, excessive agency, agent goal hijack) and deployer duties under EU AI Act Art. 26 - [ ] Seven in-scope AI/HAI software archetypes listed: LLM-integrated app, autonomous AI agent, RAG pipeline, fine-tuning/training workload, evaluation/red-team harness, model-serving service, classical ML model - [ ] Working group roster documented (Security, Engineering per product line, Data/ML platform, Privacy/Legal, Product, SRE/Platform, application-architect reviewer) - [ ] Decision rights matrix: who approves, who blocks, who handles exceptions, who owns the go-live gate - [ ] Year-one success definition with numerical targets for L1 outcome metrics
Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |---|---|---|---|---|---| | AI/HAI software inventory coverage (% of discovered AI/HAI artifacts in inventory) | ___ | ___ | ≥90% within 12 months | ☐ | | | Shadow-AI-in-engineering ratio (unsanctioned AI/HAI artifacts in production ÷ total) | ___ | ___ | ≤15% and trending down | ☐ | | | % engineering headcount covered by acknowledged AI Acceptable Use & Engineering Standards Policy | ___ | ___ | ≥95% of engineering | ☐ | | | % AI/HAI software artifacts in production with a named owning team | ___ | ___ | 100% | ☐ | | | Known data-exposure events from AI/HAI software (per quarter) | ___ | ___ | trending down QoQ | ☐ | |
Metric Collection Guidance:
- Inventory coverage: Reconcile the inventory count against all discovery-source signals (source-code, dependency-manifest, CI/CD, runtime-egress, cloud-spend). Formula: inventory_count / discovered_count × 100
- Shadow-AI-in-engineering ratio: From the inventory status field, divide artifacts with status "Provisional/Under review/Awaiting Intake/Prohibited" in production by total in-production count
- AUP attestation: Pull acknowledgment records from HR/LMS system filtered to engineering headcount; report as % acknowledged
- Named owning team: Count inventory records with a non-null owning-team field vs. total; target is 100%
- Data-exposure events: Aggregate DLP alerts, incident-tracker entries, and prompt/completion-log review findings per quarter; trend over time
Answer: - ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets) - ☐ Implemented (Evidence complete + 2 metrics meet targets) - ☐ Partial (Evidence partially complete + <2 metrics meet targets) - ☐ Not Implemented (No evidence)
Evidence Location: _____ Metric Validation Date: ____ Notes: __________
Q1.2: Do you maintain a single AI/HAI software inventory seeded from source-code signals (LLM SDK imports), dependency manifests, CI/CD telemetry, runtime egress logs, model/prompt registries, and cloud spend, covering all seven archetypes with a minimum field set including archetype, production status, data classes, approval status, and linked artifacts?
Evidence Required: - [ ] Single authoritative inventory exists with all required minimum fields populated (artifact name, owning team, archetype, production status, customer-facing flag, data classes, LLM/model providers, approval status, linked artifacts) - [ ] Source-code discovery running: grep/scanner for LLM SDK imports (openai, anthropic, langchain, transformers, vllm, bedrock, etc.) and vector-store clients across the monorepo or polyrepo - [ ] Dependency-manifest scanning active across package.json, requirements.txt, pyproject.toml, go.mod, Cargo.toml, Gemfile - [ ] CI/CD telemetry: jobs running model training, fine-tuning, or eval harnesses flagged; release pipelines flagging AI provider endpoint deployments captured - [ ] Runtime egress logs and cloud spend (Bedrock/Vertex/OpenAI/Anthropic/Azure OpenAI by team tag) used as discovery signals - [ ] Amnesty window publicized through engineering all-hands and team channels; self-disclosure carries no penalty
Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |---|---|---|---|---|---| | AI/HAI software inventory coverage (% of discovered AI/HAI artifacts in inventory) | ___ | ___ | ≥90% within 12 months | ☐ | | | Shadow-AI-in-engineering ratio (unsanctioned AI/HAI artifacts in production ÷ total) | ___ | ___ | ≤15% and trending down | ☐ | | | % AI/HAI software artifacts in production with a named owning team | ___ | ___ | 100% | ☐ | | | Known data-exposure events from AI/HAI software (per quarter) | ___ | ___ | trending down QoQ | ☐ | | | % engineering headcount covered by acknowledged AI Acceptable Use & Engineering Standards Policy | ___ | ___ | ≥95% of engineering | ☐ | |
Metric Collection Guidance: - Inventory coverage: Run discovery-source reconciliation monthly; compare inventory records to scanner output + CI/CD flags + egress logs + cloud-spend signals. Record unmatched signals as shadow-AI candidates - Shadow-AI-in-engineering ratio: Filter inventory for in-production artifacts whose approval status is not "Sanctioned"; divide by total in-production; trend quarterly - Named owning team: Automated check, inventory records where owning_team field is null; alert on new records missing owner field at creation - Data-exposure events: Combine DLP system events (sensitive data in prompt/completion logs), incident-tracker AI-related tickets, and prompt-log review findings; count per quarter - AUP attestation: LMS/HR system query for engineering headcount who completed the AI Acceptable Use acknowledgment; denominator is total engineering headcount
Answer: - ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets) - ☐ Implemented (Evidence complete + 2 metrics meet targets) - ☐ Partial (Evidence partially complete + <2 metrics meet targets) - ☐ Not Implemented (No evidence)
Evidence Location: _____ Metric Validation Date: ____ Notes: __________
Q1.3: Do you baseline and report quarterly to the executive sponsor a shadow AI scoreboard covering inventory state by archetype, new artifacts discovered and their intake status, shadow-AI-in-engineering ratio trend over the last four quarters, AUP attestation coverage, and the top five unmitigated AI-specific risks with named owners and remediation status?
Evidence Required: - [ ] Quarterly shadow AI scoreboard published and delivered to the executive sponsor, at least two consecutive quarters on record - [ ] Scoreboard includes archetype-level breakdown (LLM-integrated app, agent, RAG, fine-tune, eval harness, model-serving, classical ML) - [ ] Shadow-AI-in-engineering ratio trended over last 4 quarters with commentary on direction - [ ] AUP attestation percentage reported with engineering headcount denominator - [ ] Top 5 unmitigated AI-specific risks listed with named owner and remediation status per risk (TA-flagged, ML-flagged, or external-advisory-flagged) - [ ] Intake SLA tracked: new AI/HAI software intake triaged within 5 BD; provisional approval within 10 BD for low-risk archetypes
Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |---|---|---|---|---|---| | AI/HAI software inventory coverage (% of discovered AI/HAI artifacts in inventory) | ___ | ___ | ≥90% within 12 months | ☐ | | | Shadow-AI-in-engineering ratio (unsanctioned AI/HAI artifacts in production ÷ total) | ___ | ___ | ≤15% and trending down | ☐ | | | % engineering headcount covered by acknowledged AI Acceptable Use & Engineering Standards Policy | ___ | ___ | ≥95% of engineering | ☐ | | | % AI/HAI software artifacts in production with a named owning team | ___ | ___ | 100% | ☐ | | | Known data-exposure events from AI/HAI software (per quarter) | ___ | ___ | trending down QoQ | ☐ | |
Metric Collection Guidance: - Scoreboard delivery cadence: Confirm the last two quarters have a dated scoreboard document or deck delivered to the executive sponsor with their acknowledgment on record - Archetype breakdown: Scoreboard section showing counts per archetype (sanctioned / provisional / prohibited / awaiting intake); used to detect which archetype classes are growing unchecked - Shadow-AI-in-engineering ratio trend: Four-quarter chart or table; downward trend is the L1 success signal; source is inventory status field reconciled against discovery sweeps - AUP attestation: Reported as a percentage with the engineering headcount denominator explicitly stated; LMS or HR system is the authoritative source - Top-5 risks: Each entry lists risk description, source, named owner, and current remediation status (open / in-progress / mitigated)
Answer: - ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets) - ☐ Implemented (Evidence complete + 2 metrics meet targets) - ☐ Partial (Evidence partially complete + <2 metrics meet targets) - ☐ Not Implemented (No evidence)
Evidence Location: _____ Metric Validation Date: ____ Notes: __________
Objective: Risk-tier the AI/HAI software inventory, calibrate the program's intensity per tier, and measure practice maturity and shadow-AI reduction per tier, establishing the tier rubric every other Software-domain L2 practice depends on
Q2.1: Do you have a published risk-tier rubric (Critical / High / Medium / Low) assigning a tier to every AI/HAI software artifact based on seven auditable dimensions, data sensitivity processed, decision-affecting use (EU AI Act Annex III / GDPR Art. 22), agentic capability, user exposure, training-data posture, production-load-bearing, and concentration, with tier derivation deterministic, human overrides recorded with rationale, and 100% of inventory records carrying a current tier?
Evidence Required: - [ ] Published tier-rubric document listing all seven auditable dimensions with deterministic assignment logic - [ ] 100% of inventory records carry a current tier assignment derived from the rubric - [ ] Data sensitivity dimension covers regulated data (PHI/PCI/regulated PII/source code/customer confidential) at inference and training - [ ] Decision-affecting use dimension explicitly references EU AI Act Annex III high-risk uses and GDPR Art. 22 automated decisioning - [ ] Agentic capability dimension accounts for tool surface (executes shell, calls internal APIs, outbound HTTP, modifies records), agent scope, and tool count - [ ] Human override log maintained: all overrides recorded with rationale and reviewed by the working group at least quarterly
Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |---|---|---|---|---|---| | % of inventory with a current tier assignment | ___ | ___ | 100% | ☐ | | | Tier-treatment matrix adherence, % Critical artifacts with full-scope treatment in last 12 months | ___ | ___ | ≥95% | ☐ | | | Tier-weighted shadow AI ratio (Critical-weighted) | ___ | ___ | Critical = 0 unsanctioned in production; overall trending down | ☐ | | | Per-tier SLA adherence across practices (intake, DR, IR, ST, ML, IM) | ___ | ___ | ≥90% per tier | ☐ | | | Tier drift rate (tier changes per year) | ___ | ___ | tracked; unexplained changes = 0 | ☐ | |
Metric Collection Guidance: - % inventory with tier assignment: Automated check, count inventory records where risk_tier field is null or last-updated > 90 days without re-confirmation; flag for remediation within 5 BD - Tier-treatment matrix adherence: Cross-reference Critical-tier inventory records against evidence fields for each downstream practice (DR completed, IR completed within cadence, ST battery complete); ≥95% must show full-scope treatment in last 12 months - Tier-weighted shadow AI ratio: Filter Critical-tier in-production records not in "Sanctioned" status; this count must be 0. Overall ratio across all tiers should trend down quarter-over-quarter - Per-tier SLA adherence: From intake, DR, IR, ST, and IM trackers, calculate % of SLA-bound actions completed on time per tier; report monthly - Tier drift rate: From governance log, count tier changes per quarter; flag any change with no recorded dimension-change rationale as unexplained
Answer: - ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets) - ☐ Implemented (Evidence complete + 2 metrics meet targets) - ☐ Partial (Evidence partially complete + <2 metrics meet targets) - ☐ Not Implemented (No evidence)
Evidence Location: _____ Metric Validation Date: ____ Notes: __________
Q2.2: Do you have a published tier-treatment matrix defining differential program intensity across all downstream Software-domain practices (PC, TA, SR, SA, DR, IR, ST, EH, ML, IM) for each of the four tiers, and is this matrix enforced, with Critical-tier artifacts receiving full-scope treatment including semi-annual IR, full ST battery, all detections tuned, mandatory re-review within 14 days of material change, and IM SLA of ack ≤4h / mitigate ≤48h?
Evidence Required: - [ ] Tier-treatment matrix published covering all downstream practices with explicit treatment per tier per practice - [ ] Critical-tier treatment documented: full SR pack + executive sign-off at intake; per-artifact deep threat model with adversarial-ML overlay; full-lane DR with named architect; semi-annual IR + on material change; full ST battery; all detections tuned; IM SLA ack ≤4h/mitigate ≤48h; mandatory re-review within 14 days of material change - [ ] Low-tier fast-track documented: provisionally approved within 10 BD; no full DR required; spot-check ST; baseline logging - [ ] Evidence that Critical-tier artifacts are actually receiving full-scope treatment (artifacts cross-referenced to treatment evidence in program tracker) - [ ] Downstream practices (TA, SR, SA, DR, IR, ST, EH, ML, IM) each acknowledged the calibration in writing or via a working-group decision record
Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |---|---|---|---|---|---| | % of inventory with a current tier assignment | ___ | ___ | 100% | ☐ | | | Tier-treatment matrix adherence, % Critical artifacts with full-scope treatment in last 12 months | ___ | ___ | ≥95% | ☐ | | | Tier-weighted shadow AI ratio (Critical-weighted) | ___ | ___ | Critical = 0 unsanctioned in production; overall trending down | ☐ | | | Per-tier SLA adherence across practices (intake, DR, IR, ST, ML, IM) | ___ | ___ | ≥90% per tier | ☐ | | | Tier drift rate (tier changes per year) | ___ | ___ | tracked; unexplained changes = 0 | ☐ | |
Metric Collection Guidance: - Tier-treatment matrix adherence: Build a cross-reference table, for each Critical-tier artifact, list last DR date, last IR date, ST coverage status, ML detection status, and last IM SLA measurement; ≥95% must show all treatments completed in the last 12 months - Per-tier SLA adherence: Aggregate SLA data from intake tracker, DR queue, IR schedule, and IM system; calculate on-time completion % per tier; report monthly to the working group - Tier-weighted shadow AI ratio: Critical-tier unsanctioned artifacts must be 0; overall ratio across tiers should decrease; both reported in the quarterly scoreboard - Tier drift rate: Governance log reviewed at each working-group meeting; unexplained changes (no dimension-change rationale) flagged for sponsor visibility
Answer: - ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets) - ☐ Implemented (Evidence complete + 2 metrics meet targets) - ☐ Partial (Evidence partially complete + <2 metrics meet targets) - ☐ Not Implemented (No evidence)
Evidence Location: _____ Metric Validation Date: ____ Notes: __________
Q2.3: Does the quarterly shadow AI scoreboard report inventory state per tier and per archetype with Critical-tier unsanctioned AI in production explicitly tracked at zero, include a tier-movement log with rationale for each change, report per-tier SLA adherence, and is it reviewed by the executive sponsor who explicitly discusses tier-balance?
Evidence Required: - [ ] Quarterly scoreboard includes a tier × archetype breakdown table (Critical/High/Medium/Low rows by archetype columns) - [ ] Critical-tier unsanctioned AI in production is explicitly called out as a named metric; any non-zero value is a headline finding requiring sponsor action - [ ] Tier-movement log included: artifacts that moved up or down in the quarter, with the dimension(s) that changed and rationale for each move - [ ] SLA adherence per tier reported for intake, DR, IR, ST, ML, and IM - [ ] Quarterly executive review meeting documented (agenda + minutes) showing tier-balance discussion and sponsor sign-off - [ ] Working group sprint to onboard the next downstream practice's L2 calibration is tracked as a program activity
Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |---|---|---|---|---|---| | % of inventory with a current tier assignment | ___ | ___ | 100% | ☐ | | | Tier-treatment matrix adherence, % Critical artifacts with full-scope treatment in last 12 months | ___ | ___ | ≥95% | ☐ | | | Tier-weighted shadow AI ratio (Critical-weighted) | ___ | ___ | Critical = 0 unsanctioned in production; overall trending down | ☐ | | | Per-tier SLA adherence across practices (intake, DR, IR, ST, ML, IM) | ___ | ___ | ≥90% per tier | ☐ | | | Tier drift rate (tier changes per year) | ___ | ___ | tracked; unexplained changes = 0 | ☐ | |
Metric Collection Guidance: - Per-tier scoreboard delivery: Last two consecutive quarterly scoreboards must include a tier × archetype table, not just aggregates; each scoreboard shows delta from prior quarter - Critical-tier unsanctioned count: Explicitly named metric in each scoreboard; source is inventory filtered on tier=Critical AND status != Sanctioned AND production_status = in-production - Tier-movement log completeness: Each entry must have artifact name, prior tier, new tier, dimension(s) that changed, reviewer name, and date; unexplained changes target = 0 - SLA adherence per tier: Pulled from intake, DR, IR, ST, and IM systems; aggregated per tier; reported as % on-time per tier per quarter - Executive review minutes: Filed governance document showing the exec sponsor reviewed the tier-balance section and issued follow-up actions or signed off
Answer: - ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets) - ☐ Implemented (Evidence complete + 2 metrics meet targets) - ☐ Partial (Evidence partially complete + <2 metrics meet targets) - ☐ Not Implemented (No evidence)
Evidence Location: _____ Metric Validation Date: ____ Notes: __________
Objective: Automate inventory and tier maintenance from live build/deploy/runtime signals, benchmark the program against external peers, and contribute anonymized AI/HAI software ecosystem intelligence back to the industry
Q3.1: Does the AI/HAI software inventory auto-update from live CI/CD events, model-registry events, dependency-manifest commits, runtime egress signals, and prompt/completion log volumes, with tier assignments rule-based on versioned, replayable logic, tier changes auto-triggering downstream practice obligations within 24 hours, and a published data-quality SLO of ≥99% correctly tiered within 48 hours of a material change?
Evidence Required: - [ ] Published data-quality SLO: ≥99% of active AI/HAI artifacts correctly tiered within 48 hours of a material change; ≥95% inventory completeness against discovery-source reconciliation - [ ] Automated feeds operational: CI/CD job-type tags, model-registry events, dependency-manifest scanning on commit, runtime-egress new-flow detection, prompt/completion log volume spikes, self-attestation and intake - [ ] Tier rules documented as versioned, replayable logic, rule changes are change-logged and can be replayed against historical inventory state - [ ] Tier-change events auto-trigger downstream obligations (e.g., Medium→Critical triggers DR, ST, ML reconfiguration) within 24h; trigger success monitored via workflow telemetry - [ ] Human curation queue defined for exception cases: new archetypes, ambiguous discoveries, dimensional-input conflicts - [ ] Automation health dashboard: on-call paged when any signal feed exceeds a published staleness threshold
Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |---|---|---|---|---|---| | Inventory auto-update latency | ___ | ___ | ≤48h for material changes | ☐ | | | % inventory entries auto-curated vs. human-curated | ___ | ___ | ≥80% auto | ☐ | | | Inventory completeness against discovery-source reconciliation | ___ | ___ | ≥99% | ☐ | | | Tier-rule auto-trigger of downstream obligations on tier change | ___ | ___ | 100% within 24h | ☐ | | | External benchmarks tracked | ___ | ___ | ≥5 peer-comparable metrics | ☐ | | | Industry contributions per year | ___ | ___ | ≥4 substantive | ☐ | | | Executive ROI narrative refreshed with external benchmarks | ___ | ___ | semi-annual | ☐ | |
Metric Collection Guidance: - Auto-update latency: Measure time from a known material change event (new LLM SDK import merged, new model deployed) to the corresponding inventory record update; sample 20 events per quarter; report P95 latency - % auto-curated: From the inventory curation log, count records updated by automated feeds vs. human-initiated edits; report as a ratio per quarter - Inventory completeness: Run the full discovery-source reconciliation (scanner + CI/CD + egress + registries + cloud-spend) and compare to inventory count; report gap as completeness % - Downstream obligation auto-trigger: From workflow telemetry, verify each tier-change event produced a downstream obligation trigger (DR ticket, ST job, ML reconfiguration) within 24h; report % with trigger within SLO - External benchmarks tracked: Count distinct external benchmark data points in the semi-annual brief; each must be traceable to a named source
Answer: - ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets) - ☐ Implemented (Evidence complete + 2 metrics meet targets) - ☐ Partial (Evidence partially complete + <2 metrics meet targets) - ☐ Not Implemented (No evidence)
Evidence Location: _____ Metric Validation Date: ____ Notes: __________
Q3.2: Do you publish a semi-annual external-benchmarking brief comparing the program against at least five peer-comparable metrics via OWASP SAMM AI extensions, OpenSSF AI, MITRE ATLAS practitioner data exchanges, sector ISACs with AI working groups, and formal peer roundtables, and do benchmark deltas explicitly inform program investment decisions?
Evidence Required: - [ ] Semi-annual benchmarking brief published, two most recent on file with dates, each containing ≥5 peer-comparable metrics from named external sources - [ ] Benchmarking sources include at least two of: OWASP SAMM AI / OpenSSF AI / MITRE ATLAS practitioner exchanges / sector ISAC AI working groups / BSIMM-style observational data / formal peer roundtables - [ ] Metrics benchmarked cover program-relevant dimensions: inventory coverage, shadow-AI ratio, per-tier SLA adherence, automation level, time-from-intake-to-provisional-approval - [ ] Benchmark deltas explicitly referenced in a program investment or prioritization decision with documentation filed within 90 days of each brief - [ ] Peer selection rationale documented, peers chosen to stretch the program, not flatter it
Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |---|---|---|---|---|---| | Inventory auto-update latency | ___ | ___ | ≤48h for material changes | ☐ | | | % inventory entries auto-curated vs. human-curated | ___ | ___ | ≥80% auto | ☐ | | | Inventory completeness against discovery-source reconciliation | ___ | ___ | ≥99% | ☐ | | | Tier-rule auto-trigger of downstream obligations on tier change | ___ | ___ | 100% within 24h | ☐ | | | External benchmarks tracked | ___ | ___ | ≥5 peer-comparable metrics | ☐ | | | Industry contributions per year | ___ | ___ | ≥4 substantive | ☐ | | | Executive ROI narrative refreshed with external benchmarks | ___ | ___ | semi-annual | ☐ | |
Metric Collection Guidance: - External benchmarks tracked: Each semi-annual brief lists ≥5 named benchmark data points; each is traceable to an ISAC report, OWASP publication, BSIMM dataset, or named peer roundtable - Benchmark-driven investment: Look for a program planning or budget document that explicitly cites a benchmark delta as a rationale; filed within 90 days of each benchmarking brief - Semi-annual cadence: Verify two briefs within a 12-month window; no gap > 7 months between consecutive briefs - Executive ROI narrative: Annual exec/board briefing deck or memo includes benchmark comparisons alongside program metrics and documents avoided-loss examples
Answer: - ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets) - ☐ Implemented (Evidence complete + 2 metrics meet targets) - ☐ Partial (Evidence partially complete + <2 metrics meet targets) - ☐ Not Implemented (No evidence)
Evidence Location: _____ Metric Validation Date: ____ Notes: __________
Q3.3: Does the program contribute at least four substantive, anonymized artifacts per year to the AI/HAI software assurance ecosystem through MITRE ATLAS, OWASP LLM / Agentic Top 10, NIST AI RMF Playbook, AVID, OpenSSF AI, or sector ISACs, with each contribution anonymized, legally vetted, and traceable to a published working-group output or standard?
Evidence Required: - [ ] Contribution log maintained listing all submissions: target body, submission type, date submitted, anonymization review completed, and status (in-review / accepted / published) - [ ] At least 4 substantive contributions per year in the most recent 12-month window; each is a technical artifact accepted or in active review by the named body - [ ] Each contribution has a legal/privacy review sign-off confirming anonymization before submission - [ ] Contributions traceable to published outputs: MITRE ATLAS entries, OWASP review comments incorporated, NIST AI RMF chapter references, AVID entries, OpenSSF advisory publications - [ ] Contribution pipeline shows ≥2 items in-flight (draft, in-review, or being prepared) at any working-group review
Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |---|---|---|---|---|---| | Inventory auto-update latency | ___ | ___ | ≤48h for material changes | ☐ | | | % inventory entries auto-curated vs. human-curated | ___ | ___ | ≥80% auto | ☐ | | | Inventory completeness against discovery-source reconciliation | ___ | ___ | ≥99% | ☐ | | | Tier-rule auto-trigger of downstream obligations on tier change | ___ | ___ | 100% within 24h | ☐ | | | External benchmarks tracked | ___ | ___ | ≥5 peer-comparable metrics | ☐ | | | Industry contributions per year | ___ | ___ | ≥4 substantive | ☐ | | | Executive ROI narrative refreshed with external benchmarks | ___ | ___ | semi-annual | ☐ | |
Metric Collection Guidance: - Industry contributions per year: Count entries in the contribution log for trailing 12 months where status = submitted or accepted to a named body; conference talks and press releases do not count - Contribution pipeline health: At any working-group meeting, the pipeline log should show ≥2 items not yet in submitted status; checked at each working-group meeting and noted in minutes - Legal/privacy review: Each contribution log entry must have a legal/privacy reviewer name and date; no contribution submitted without this sign-off - Executive ROI narrative: Filed annually to exec/board; references external benchmarks and lists avoided-loss examples (incidents prevented, regulatory exposure mitigated, faster sanctioned ship time)
Answer: - ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets) - ☐ Implemented (Evidence complete + 2 metrics meet targets) - ☐ Partial (Evidence partially complete + <2 metrics meet targets) - ☐ Not Implemented (No evidence)
Evidence Location: _____ Metric Validation Date: ____ Notes: __________
| Level | Q1 | Q2 | Q3 | Avg | Achieved? |
|---|---|---|---|---|---|
| L1 | __ | __ | __ | __ | ☐ |
| L2 | __ | __ | __ | __ | ☐ |
| L3 | __ | __ | __ | __ | ☐ |
Practice maturity level achieved: ___ (highest level where all 3 questions score ≥ 0.67)
Document Version: HAIAMM v3.0 Practice: Strategy & Metrics (SM) Domain: Software Last Updated: 2026-05-15 Author: Verifhai
Instructions: