Loading...
AI Model Poisoning
AMPInjection of malicious data into AI training datasets to corrupt model behavior, causing models to learn incorrect patterns or exhibit harmful behaviors.
Example Scenario
NullBulge ransomware group targets Hugging Face and GitHub repositories, poisoning datasets to compromise AI models during training phase, affecting thousands of downstream applications.
Testing Objectives
- Test data validation mechanisms
- Assess training pipeline security
- Evaluate data source integrity
- Validate model behavior consistency
Defensive Strategies
- Data validation and sanitization
- Source verification and digital signatures
- Statistical anomaly detection
- Multi-source data verification
- Secure data storage and access controls
Key Features
- Training data manipulation
- Gradual behavior modification
- Trigger-based activation
- Stealthy persistence
Use Cases
- Training data integrity testing
- Data validation pipeline assessment
- Supply chain security evaluation
- Model robustness verification
Tools & Frameworks
Security Risks
Ethical Guidelines
- •Only test on datasets you own or have permission to modify
- •Never poison production training data
- •Report vulnerabilities in data pipelines responsibly
- •Focus on defensive testing, not offensive poisoning
- •Consider impact on AI safety and reliability
Remember: This information is for educational and defensive security purposes only. Always ensure you have proper authorization before testing any techniques.
From the engineer behind this catalog
Get your agent system red-teamed
The attacks documented here work on production agent systems every day. Have yours tested before someone else does: prompt injection, jailbreaks, tool misuse and data exfiltration, with every finding written up next to its fix.
€750 instead of €1,500, one week, written report and walkthrough call, until 30 September