AI Security Checklist: 50+ Controls for Secure AI Applications

Artificial intelligence is becoming part of everyday software development and business operations. Applications now use Large Language Models (LLMs), retrieval-augmented generation (RAG), AI agents, recommendation systems, and machine learning models to process information and perform tasks.

However, integrating AI into an application introduces security risks that traditional application security controls may not fully address. An AI application can expose sensitive information through its responses, follow malicious instructions embedded in external content, retrieve documents a user should not access, or perform unauthorized actions through connected tools and APIs.

Security should therefore be considered throughout the AI application lifecycle, from design and data preparation to deployment, monitoring, and maintenance.

This AI security checklist provides practical controls that developers, application security engineers, cloud security teams, and architects can use when designing, reviewing, or assessing AI-powered applications. It covers governance, data protection, model security, prompt injection, RAG, AI agents, infrastructure, monitoring, and security testing.

1. AI Governance and Risk Management

Before implementing technical safeguards, organizations need to understand where AI is being used, what information it processes, and what decisions it can influence. Without clear ownership and visibility, security teams may overlook AI services introduced through third-party integrations or individual development projects.

  • Maintain an inventory of AI applications, models, third-party AI services, APIs, datasets, and connected tools.
  • Document the purpose of each AI system and identify its intended users.
  • Classify AI use cases according to their potential business, privacy, and security impact.
  • Assign clear ownership for AI security, privacy, development, and operational responsibilities.
  • Define acceptable-use requirements for employees and developers using external AI services.
  • Document what information can and cannot be submitted to public or third-party AI platforms.
  • Review applicable privacy, contractual, regulatory, and organizational requirements.
  • Include AI-related threats and dependencies in the organization's risk assessment process.
  • Define procedures for reporting AI security incidents and unsafe model behavior.

What to verify: Can the organization identify every AI service used in production? Is there an assigned owner for each system? Can the security team determine what data the system processes and which external providers receive it?

2. Data Security and Privacy

AI systems often process large amounts of information, including customer records, internal documents, source code, business information, and personal data. Data exposure can occur during training, inference, retrieval, logging, or integration with external services.

Data security controls should cover the complete data lifecycle rather than focusing only on database encryption.

  • Classify information before using it for model training, fine-tuning, retrieval, or inference.
  • Remove or mask unnecessary personally identifiable information (PII) and confidential data.
  • Validate the source, integrity, and quality of training datasets.
  • Restrict access to datasets, embeddings, vector databases, and data-processing pipelines.
  • Protect training and ingestion pipelines against unauthorized modification.
  • Define data retention and deletion requirements for prompts, responses, and uploaded files.
  • Understand whether third-party AI providers retain submitted data or use it for model training.
  • Encrypt sensitive information in transit and at rest.
  • Prevent sensitive information from appearing in application logs, debugging output, and error messages.
  • Apply data access restrictions before retrieving information for AI-generated responses.

Why access control matters in RAG applications

Retrieval-Augmented Generation (RAG) allows an AI application to retrieve relevant information from external sources, such as internal documents, knowledge bases, and databases. The retrieved information is then provided to the model as context for generating an answer.

A common security mistake is to assume that because a user cannot directly access a document through the normal application interface, the AI system cannot expose it either. If document-level permissions are not enforced during retrieval, the model may receive confidential information and include it in its response.

For example, an employee using an internal AI assistant should not be able to retrieve HR records or financial documents simply by asking the model. Authorization must be enforced by the backend retrieval service before documents are supplied to the model.

What to verify: Test whether users with different roles can retrieve the same documents. Confirm that unauthorized records are excluded before they reach the model, not merely hidden in the final response.

3. Model and AI Supply Chain Security

AI applications depend on more than the model itself. They may use pretrained models, fine-tuning datasets, model repositories, libraries, plugins, external APIs, and inference frameworks. Each dependency can introduce security and integrity risks.

  • Use trusted model repositories and verify model provenance where possible.
  • Review model licenses, supported versions, and security-related documentation.
  • Assess third-party models and AI services before integrating them into production.
  • Restrict access to model files, model configurations, and deployment pipelines.
  • Scan application dependencies and AI frameworks for known vulnerabilities.
  • Maintain version control for models, datasets, prompts, and configurations.
  • Protect model artifacts against unauthorized replacement or modification.
  • Review external plugins, extensions, and tools before granting them access to AI applications.
  • Maintain an inventory of critical AI dependencies and their owners.
  • Establish a process for updating or replacing vulnerable models and dependencies.

What to verify: Can the team identify where its model and dependencies came from? Are model updates controlled and traceable? Can unauthorized users replace a deployed model or modify its configuration?

4. Prompt and Input Security

Prompt injection is an important security risk in LLM-powered applications. It occurs when malicious instructions are introduced through user input or external content in an attempt to manipulate the model's behavior.

Direct prompt injection comes from instructions supplied directly by a user. Indirect prompt injection occurs when malicious instructions are embedded in content that an AI system processes, such as webpages, documents, emails, or retrieved knowledge-base records.

  • Test for direct and indirect prompt injection attacks.
  • Treat user prompts and retrieved external content as untrusted input.
  • Keep system instructions separate from user-controlled content in the application design.
  • Do not rely on system prompts as the only security boundary.
  • Set appropriate limits on prompt length, request size, and input frequency.
  • Validate uploaded files and restrict unsupported file types.
  • Apply input filtering where it provides meaningful protection.
  • Prevent untrusted content from directly controlling privileged application operations.
  • Test whether malicious instructions can override intended application behavior.
  • Apply rate limits and abuse controls to publicly accessible AI endpoints.

Example: Indirect prompt injection

Consider an AI assistant that summarizes documents uploaded by users. An attacker uploads a document containing hidden instructions telling the assistant to reveal information from its previous context or ignore its normal restrictions.

If the application treats document content as trusted instructions, the model may follow those instructions while processing the file.

The application should treat uploaded documents as untrusted data, restrict the information supplied to the model, and prevent model-generated instructions from directly triggering privileged operations.

What to verify: Test malicious instructions in user prompts, uploaded documents, webpages, and retrieved content. Check whether the model can be manipulated into exposing data or initiating actions outside its intended purpose.

5. Output Security and Response Validation

AI-generated responses should not automatically be treated as safe or trustworthy. A model may produce incorrect information, disclose sensitive data, generate unsafe code, or return content that becomes dangerous when processed by another application component.

  • Validate AI-generated output before using it in application workflows.
  • Escape or sanitize output before rendering it in HTML or other executable contexts.
  • Prevent generated content from being executed as SQL, shell commands, or application code without appropriate validation.
  • Check that responses do not contain unauthorized personal or confidential information.
  • Validate AI-generated code through established code review and security testing processes.
  • Apply schema validation when the model is expected to return structured data.
  • Do not use model-generated authorization decisions as a replacement for backend access controls.
  • Require human review for high-impact decisions and sensitive operations.
  • Provide clear error handling when the model returns invalid or unexpected output.

What to verify: Can a model response containing HTML, JavaScript, SQL, or shell commands cause unintended behavior in the application? Are generated actions and structured responses validated independently?

6. RAG and Vector Database Security

RAG systems introduce additional security considerations because they connect language models to external knowledge sources. Security weaknesses may exist in document ingestion, embedding generation, vector storage, retrieval, and response generation.

  • Enforce authorization checks before retrieving documents from a knowledge base.
  • Apply tenant and user isolation in multi-tenant RAG applications.
  • Validate documents before adding them to the knowledge base.
  • Restrict who can upload, modify, and delete indexed content.
  • Protect embedding services and vector databases with authentication and network restrictions.
  • Prevent unauthorized access to vector database collections and metadata.
  • Monitor changes to indexed documents and embedding records.
  • Test for malicious instructions embedded in retrieved documents.
  • Define procedures for removing outdated, malicious, or unauthorized content.
  • Ensure retrieved context does not bypass the application's existing data access policies.

What to verify: Attempt cross-user and cross-tenant retrieval. Test whether a user can access documents by manipulating queries or metadata. Confirm that removing a document from the source also removes or invalidates its indexed representation where required.

7. AI Agent and Tool Security

AI agents introduce a different level of risk because they can interact with external systems and perform actions. Depending on their configuration, agents may call APIs, execute code, access files, query databases, send emails, or trigger business workflows.

A compromised or manipulated agent may therefore cause real operational impact rather than simply generate an unsafe response.

  • Give agents only the permissions required for their specific tasks.
  • Restrict access to tools, APIs, files, and external services.
  • Enforce authorization in the tool or backend service, not only in the model prompt.
  • Require explicit human approval for sensitive or irreversible actions.
  • Validate tool arguments before executing operations.
  • Use separate credentials and identities for individual agents where appropriate.
  • Protect API keys, tokens, and secrets from model context and generated responses.
  • Restrict agent access to sensitive systems and administrative functions.
  • Maintain audit logs of tool calls, actions, and execution results.
  • Test whether prompt injection can cause unauthorized tool execution.
  • Set execution limits, timeouts, and safeguards for automated workflows.
  • Define a process to disable an agent or revoke its permissions during an incident.

Example: An AI agent with email access

Imagine an AI assistant that can read incoming emails and send replies. An attacker sends an email containing malicious instructions intended to manipulate the agent into forwarding confidential messages to an external address.

The agent should not have unrestricted permission to forward messages. The application should enforce recipient restrictions, validate the requested action, and require approval for sensitive data transfers.

What to verify: Test whether an agent can access tools outside its intended scope, execute actions without approval, or use manipulated input to perform unauthorized operations.

8. Infrastructure, API, and Cloud Security

AI applications rely on infrastructure that includes cloud services, containers, APIs, model-serving endpoints, storage systems, and networking components. These systems should be protected using established application and cloud security practices.

  • Protect AI APIs with strong authentication and authorization.
  • Apply rate limiting, quotas, and request-size restrictions.
  • Use network segmentation and restrict access to internal model endpoints.
  • Encrypt sensitive information in transit and at rest.
  • Store API keys and service credentials in an approved secrets management solution.
  • Apply least-privilege IAM policies to AI workloads and service identities.
  • Secure container images, runtime environments, and orchestration platforms.
  • Patch operating systems, AI frameworks, libraries, and infrastructure components.
  • Use secure configuration baselines for cloud resources and deployment environments.
  • Separate development, testing, and production AI environments.
  • Protect model-serving infrastructure against denial-of-service and resource exhaustion attacks.
  • Review third-party API permissions and external network connections.

What to verify: Check whether AI endpoints are publicly exposed unnecessarily, whether credentials are stored securely, and whether workloads have excessive cloud permissions. Review API authentication, network access, and resource limits.

9. Logging, Monitoring, and Incident Response

AI security monitoring should cover both traditional infrastructure activity and AI-specific behavior. A system may be operating normally from an infrastructure perspective while experiencing prompt injection, data extraction attempts, automated abuse, or unauthorized tool usage.

  • Enable audit logging for AI application activity and administrative changes.
  • Record relevant model requests, responses, tool calls, and execution outcomes while protecting sensitive data.
  • Monitor unusual request volumes and abnormal token consumption.
  • Detect repeated prompt injection attempts and suspicious input patterns.
  • Monitor access to sensitive documents and vector database collections.
  • Alert on unexpected tool usage and privileged operations.
  • Monitor API errors, latency, resource consumption, and unusual cost increases.
  • Establish procedures for investigating suspected AI security incidents.
  • Define how to disable affected integrations, revoke credentials, or restrict model access.
  • Review security events and update detection rules based on observed threats.

Logging should be designed carefully. Storing complete prompts and responses can create another sensitive data repository. Apply access restrictions, retention limits, and masking where appropriate.

What to verify: Can the security team identify who accessed an AI service, which tools were invoked, and whether sensitive resources were accessed? Can suspicious activity generate an alert and trigger an investigation?

10. AI Security Testing and Secure Development

AI security testing should be integrated into the Software Development Life Cycle (SDLC). Traditional application security testing remains important, but it should be supplemented with tests that address model behavior, prompt injection, data leakage, retrieval permissions, and agent capabilities.

  • Perform threat modeling before introducing AI features or changing agent capabilities.
  • Test for direct and indirect prompt injection.
  • Assess sensitive information disclosure and unintended data retrieval.
  • Test authorization controls across users, roles, and tenants.
  • Assess model and API endpoints for abuse and resource exhaustion.
  • Review training data and ingestion pipelines for poisoning risks.
  • Test RAG retrieval boundaries and vector database access restrictions.
  • Test AI agent tool permissions and approval workflows.
  • Perform SAST, SCA, secrets scanning, and container security assessments where applicable.
  • Conduct security reviews of AI integrations and third-party services.
  • Include AI-specific security tests in CI/CD and release processes where practical.
  • Retest after significant model, prompt, dataset, tool, or application changes.

Example AI security test scenarios

A practical assessment should include scenarios that reflect how the application is actually used. For example:

  1. Prompt injection: Can a malicious user manipulate the model into ignoring its intended instructions?
  2. Sensitive information disclosure: Can one user obtain another user's private information through AI-generated responses?
  3. RAG authorization: Can the retrieval service return documents that the requesting user is not authorized to access?
  4. Excessive agency: Can an agent execute an administrative action without the required approval?
  5. Unsafe output handling: Can generated HTML, code, or structured data trigger unintended behavior in the application?
  6. Resource abuse: Can repeated or oversized requests cause excessive token consumption or service disruption?
  7. Supply-chain integrity: Can an unauthorized model, dependency, or dataset be introduced into the deployment process?

Record each finding with the affected component, attack scenario, security impact, evidence, and recommended remediation. Retest the issue after the fix to confirm that the control works as expected.

11. Human Oversight and Continuous Improvement

AI systems can behave differently when models, prompts, datasets, or external dependencies change. Security controls that worked during initial testing may become ineffective after an update or integration change.

  • Require human approval for high-impact or irreversible AI-driven actions.
  • Provide users with a way to report inaccurate, unsafe, or suspicious AI responses.
  • Review model behavior and security findings periodically.
  • Update threat models when new features, tools, or data sources are introduced.
  • Reassess risks after model upgrades, fine-tuning, and configuration changes.
  • Maintain regression tests for previously identified AI security weaknesses.
  • Review relevant security guidance and emerging attack techniques.
  • Track remediation actions and assign owners for unresolved risks.

Continuous improvement is particularly important for AI agents and RAG applications because their behavior depends on changing tools, data sources, permissions, and external services.

12. Final AI Application Security Checklist

Use the following summary when reviewing an AI application before production deployment. A checked item should represent a control that has been implemented and verified, not simply discussed or planned.

Governance

  • ☐ AI systems, models, services, and owners are inventoried.
  • ☐ AI use cases and associated risks are documented.
  • ☐ Security and privacy responsibilities are assigned.
  • ☐ Acceptable-use and data-handling requirements are defined.

Data and Privacy

  • ☐ Sensitive data is classified and protected.
  • ☐ Unnecessary personal and confidential information is removed or masked.
  • ☐ Training and retrieval data sources are validated.
  • ☐ Data access, retention, and deletion controls are enforced.

Model and Supply Chain

  • ☐ Model sources and dependencies are reviewed.
  • ☐ Model artifacts and configurations are protected.
  • ☐ Model and dependency versions are tracked.
  • ☐ Third-party AI services are assessed before integration.

Prompt and Output Security

  • ☐ Direct and indirect prompt injection are tested.
  • ☐ Untrusted content is not treated as trusted instructions.
  • ☐ Model outputs are validated and safely handled.
  • ☐ Sensitive information disclosure is tested.

RAG and AI Agents

  • ☐ Retrieval permissions are enforced before context reaches the model.
  • ☐ Vector databases and knowledge sources are protected.
  • ☐ Agent tools follow least-privilege access.
  • ☐ Sensitive agent actions require appropriate approval.

Infrastructure and Monitoring

  • ☐ APIs and model endpoints are authenticated and rate-limited.
  • ☐ Cloud permissions and network access are restricted.
  • ☐ Secrets are stored securely.
  • ☐ AI activity and privileged tool calls are logged.
  • ☐ Alerts and incident response procedures are in place.

Testing and Maintenance

  • ☐ AI-specific security testing is included in the SDLC.
  • ☐ Threat models are reviewed after significant changes.
  • ☐ Security findings are tracked and retested.
  • ☐ Regression testing is performed after model and application updates.

Conclusion

Securing an AI application requires more than protecting the model endpoint or adding restrictions to a system prompt. Security teams must consider the data an AI system can access, the instructions it processes, the tools it can invoke, and the infrastructure supporting its operation.

This AI security checklist provides a starting point for reviewing those areas before production deployment. Not every control will apply to every AI system. The appropriate safeguards depend on the application's architecture, data sensitivity, users, business impact, and level of autonomy.

Use the checklist alongside threat modeling, secure software development practices, and application-specific security testing. Revisit it whenever the model, data sources, integrations, or capabilities change.

Further Reading and Security Resources

Comments