AI Risk Assessment Framework

This skill helps you assess and document security risks in AI/ML systems using industry-standard frameworks: OWASP Top 10 ML, Google SAIF, MITRE ATLAS, and LLMJacking patterns.

When to Use This Skill

Use this skill when:

• You need to identify security vulnerabilities in an AI/ML system
• You're conducting a security audit or threat modeling session
• You need to document AI risks for compliance or stakeholder review
• You're designing security controls for an AI system
• You want to understand specific attack vectors (prompt injection, data poisoning, model theft, etc.)
• You need mitigation strategies for identified AI risks

Quick Reference: Risk Frameworks

OWASP Top 10 ML Vulnerabilities

#	Vulnerability	What It Is	Example
1	Input Manipulation	Tiny changes to input data fool the model	Paint specks on stop sign → speed limit sign
2	Data Poisoning	Training data polluted with bad samples	Malware labeled as benign in antivirus training
3	Model Inversion	Reconstruct sensitive inputs from outputs	Rebuild patient MRI from cancer model predictions
4	Membership Inference	Detect if specific record was in training	Confirm bank transaction in fraud model training data
5	Model Theft	Clone model behavior via repeated queries	Harvest Q&A pairs to build equivalent local model
6	AI Supply-Chain	Compromise ML pipeline components	Poisoned dependency installs backdoored model
7	Transfer Learning Attack	Malicious logic survives fine-tuning	Vision backbone with hidden trigger persists after adaptation
8	Model Skewing	Biased data shifts outputs to attacker's agenda	Spam emails labeled as ham to bypass filter
9	Output Integrity	Alter predictions in transit	Flip "malicious" verdict to "benign" before quarantine
10	Model Poisoning	Direct changes to model parameters	Tweak fraud detection weights to approve certain cards

Google SAIF Risks

Risk	Description
Data Poisoning	Malicious actors alter training/tuning data to degrade accuracy or implant backdoors
Unauthorized Training Data	Ingesting copyrighted, sensitive, or unpermitted datasets creates legal/ethical liabilities
Model Source Tampering	Supply-chain manipulation embeds hidden logic that persists after retraining
Excessive Data Handling	Weak retention controls store more personal data than necessary
Model Exfiltration	Attackers steal model files/weights, causing IP loss
Model Deployment Tampering	Adversaries modify model artifacts so running model differs from vetted version
Denial of ML Service	Flooding APIs or "sponge" inputs exhaust compute and knock model offline
Model Reverse Engineering	Harvesting input-output pairs to clone or distil the model
Insecure Integrated Component	Vulnerable plugins/agents let attackers inject code or escalate privileges
Prompt Injection	Crafting prompts to override system intent and perform unintended commands
Model Evasion	Designed inputs trigger mis-classification, hallucination, or disallowed content
Sensitive Data Disclosure	Model reveals private/confidential information from training or user context
Inferred Sensitive Data	Model deduces personal attributes never provided, creating privacy harms
Insecure Model Output	Unsanitized responses pass harmful code, misinformation, or inappropriate content
Rogue Actions	Autonomous agents execute unintended real-world operations without oversight

MITRE AI ATLAS Matrix

The MITRE ATLAS Matrix provides a comprehensive framework for understanding AI attack techniques and tactics. It covers:

• How adversaries attack AI models
• How adversaries use AI systems to perform attacks

Reference: https://atlas.mitre.org/matrices/ATLAS

LLMJacking (Token Theft & Resale)

What it is: Attackers steal active session tokens or cloud API credentials and invoke paid, cloud-hosted LLMs without authorization. Access is resold via reverse proxies.

Consequences:

• Financial loss from unauthorized usage
• Model misuse outside policy
• Attribution to victim tenant

TTPs (Tactics, Techniques, Procedures):

• Harvest tokens from infected developer machines or browsers
• Steal CI/CD secrets; buy leaked cookies
• Stand up reverse proxy that forwards requests to genuine provider
• Abuse direct base-model endpoints to bypass enterprise guardrails

Mitigations:

• Bind tokens to device fingerprint, IP ranges, and client attestation
• Enforce short expirations and refresh with MFA
• Scope keys minimally (no tool access, read-only where applicable)
• Rotate keys on anomaly detection
• Terminate all traffic server-side behind a policy gateway
• Monitor for unusual usage patterns (spend spikes, atypical regions, UA strings)
• Prefer mTLS or signed JWTs over long-lived static API keys

Assessment Workflow

Step 1: Identify the System Type

Determine what kind of AI system you're assessing:

• ML Model (classification, regression, clustering)
• LLM/GenAI (chatbot, code assistant, content generator)
• AI Pipeline (data ingestion → training → deployment)
• AI Agent (autonomous system with tool access)

Step 2: Map to Frameworks

Use the appropriate framework(s) based on system type:

System Type	Primary Framework	Secondary Frameworks
ML Model	OWASP Top 10 ML	Google SAIF
LLM/GenAI	Google SAIF	OWASP Top 10 ML
AI Pipeline	MITRE ATLAS	OWASP Top 10 ML, Google SAIF
AI Agent	Google SAIF	MITRE ATLAS
Cloud LLM Access	LLMJacking patterns	Google SAIF

Step 3: Conduct Risk Assessment

For each relevant risk category:

1. Identify - Does this risk apply to your system?
2. Assess - What's the likelihood and impact?
3. Document - Record findings with evidence
4. Mitigate - Apply appropriate controls

Step 4: Document Findings

Use this structure for risk documentation:

## Risk: [Risk Name]

**Framework:** [OWASP/SAIF/ATLAS/LLMJacking]

**Description:** [What the risk is]

**Applicability:** [Why it applies to this system]

**Likelihood:** [Low/Medium/High]

**Impact:** [Low/Medium/High]

**Evidence:** [Specific observations, test results, or analysis]

**Mitigation:** [Recommended controls]

**Status:** [Open/Mitigated/Accepted]

Common Mitigation Patterns

Data Security

• Implement data validation and sanitization at ingestion
• Use differential privacy for training data
• Encrypt data at rest and in transit
• Implement access controls and audit logging
• Regular data quality audits

Model Security

• Model signing and integrity verification
• Secure model storage with access controls
• Model versioning and rollback capabilities
• Adversarial training and robustness testing
• Model output validation and filtering

API Security

• Rate limiting and quota enforcement
• Input validation and prompt filtering
• Output sanitization
• Authentication and authorization
• Request/response logging

Infrastructure Security

• Network segmentation for AI components
• Secure CI/CD pipelines
• Dependency scanning and verification
• Container security for model serving
• Monitoring and alerting

Quick Assessment Checklist

Use this checklist for rapid risk identification:

• Are training data sources verified and authorized?
• Is there input validation on all model inputs?
• Are model outputs sanitized before use?
• Are API keys and credentials properly secured?
• Is there rate limiting on model endpoints?
• Are there monitoring and alerting for anomalous behavior?
• Is there a process for model versioning and rollback?
• Are dependencies scanned for vulnerabilities?
• Is there access control on model artifacts?
• Are there safeguards against prompt injection (for LLMs)?
• Is there protection against model theft/exfiltration?
• Are there controls to prevent rogue agent actions?

Next Steps

After completing your assessment:

1. Prioritize risks by likelihood and impact
2. Create a remediation plan with timelines
3. Implement mitigations in order of priority
4. Test that mitigations are effective
5. Document the security posture for stakeholders
6. Schedule regular reassessments (quarterly recommended)

References

• OWASP Top 10 Machine Learning Security
• Google SAIF - Secure AI Framework
• MITRE ATLAS Matrix
• Unit 42 – The Risks of Code Assistant LLMs
• LLMJacking scheme overview