AI Security Checklist for Applications
A simple checklist to refer before deploying any AI based application into production/general use.
Governance
☐ Clearly define the usage of AI in the privacy policy or create a seperate AI security policy.
☐ Assign clear accountability and security responsibilities.
☐ Maintain an inventory of usage AI models(in-built/third party), use of datasets, prompts, and external AI services.
☐ Ensure compliance with organizational and regulatory requirements, specifically keep an eye on developing regulatory requirements.
Data Security
☐ Classify confidential,sensitive data before using it with AI.
☐ Remove/mask unnecessary personal(PII) or confidential information.
☐ Validate the source and integrity of training data.
☐ Protect data pipelines/algorithms from unauthorized modification.
Model Security
☐ Restrict access to AI models.
☐ Use authentication and authorization for model endpoints and configurations.
☐ Monitor the model abuse and unusual behavior by keeping track of usage logs.
☐ Maintain version control for models.
Prompt & Input Security
☐ Validate all user input.
☐ Protect against prompt injection attacks(direct/indirect).
☐ Separate trusted system prompts from user input.
☐ Limit excessive prompt length.
☐ Apply input filtering where appropriate.
Output Security
☐ Apply Human review stage for AI generated responses before any critical actions.
☐ Prevent disclosure of sensitive information.
☐ Validate AI generated code before deployment.
☐ Escape output before displaying it in applications.
Infrastructure Security
☐ Secure APIs using authentication and rate limiting.
☐ Encrypt data in transit and at rest.
☐ Apply least-privilege access.
☐ Keep AI frameworks and dependencies up to date.
Monitoring
☐ Enable audit logging.
☐ Monitor token usage and API costs.
☐ Detect abnormal prompt patterns, model behaviours.
☐ Alert on suspicious AI activity.
Testing
☐ Test for prompt injection.
☐ Test for sensitive data leakage.
☐ Test authorization controls.
☐ Test model abuse scenarios.
☐ Include AI specific security testing in the SDLC.(testing phase)
Human Oversight
☐ Require human approval for high-risk actions.
☐ Provide users with a method to report incorrect or unsafe AI responses.
☐ Regularly review AI behavior and update safeguards.
Continuous Improvement
☐ Periodically reassess AI risks.
☐ Update threat models as needed.
☐ Review new OWASP AI documents for guidance and be upto date.
☐ Continous retesting after major model or application changes.