AI agents can connect language models to the real world, turning instructions into actions like sending emails or accessing databases. This power creates new security challenges. Effective AI agent security isn't about perfectly instructing a model; it's about building a system around the model that enforces rules and limits what the agent can possibly do.

Key takeaways

  • Enforce authorization outside the model. System prompts and instructions are not a reliable security boundary and can be bypassed.
  • Apply the Principle of Least Privilege (PoLP). Grant an agent only the specific permissions and tools it needs for a single task, for the shortest time possible.
  • Protect memory and context. Documents, websites, and tool outputs can be used for indirect prompt injection or memory poisoning attacks that hijack the agent's goal.
  • Give agents their own identity. Separate an agent's identity from the user's identity to prevent it from inheriting excessive permissions.
  • Require human approval for high-impact actions. For irreversible actions like financial transactions, force a human review, but verify the action's details independently of the model's summary.

What is AI Agent Security and Why Does it Matter?

AI agent security is the practice of controlling and validating what an agent can access and do with its tools and data. It involves enforcing permissions with deterministic code outside the model itself. This is critical because agents can translate vague natural language into specific, real-world actions, creating new ways for systems to be attacked.

Unlike traditional software with predictable inputs, an AI agent's behavior is influenced by its instructions, tools, and the data it processes. This introduces unique risks. The Open Web Application Security Project (OWASP) highlights agent goal hijacking, tool misuse, and identity abuse as primary risks for agentic applications. An insecure agent with access to APIs could be tricked into deleting data, exposing private information, or making unauthorized purchases.

Simply telling an agent in its prompt "do not delete files" is not security. True security comes from an external system that denies the agent the technical ability to delete files in the first place.

A Threat Model for AI Agents: From Prompt to Side Effect

To secure an agent, you must intercept and validate its plans before they become actions. A robust security model separates the language model's probabilistic planning from the system's deterministic execution. The model can propose an action, but a separate service must approve it based on strict rules.

This separation is achieved with a policy-enforcement gateway, a concept also known as a Policy Enforcement Point (PEP). This gateway is a security layer that sits between the agent's brain (the LLM) and its hands (the tools). It inspects every action the agent wants to take and checks it against a list of rules before allowing it to proceed.

Here is a step-by-step model of a secure agent workflow:

  1. Input: The user provides a goal or prompt.
  2. Context Augmentation: The agent retrieves relevant information from its memory or external sources like documents and websites. This is a common entry point for indirect prompt injection, where malicious data can hide in retrieved content. Protecting this context from being poisoned is crucial.
  3. Model Plan: The Large Language Model (LLM) processes the input and context, forming a plan. This plan includes a request to use a specific tool with certain parameters (e.g., send_email(to='...', subject='...')).
  4. Policy Interception: The policy-enforcement gateway intercepts this tool call request. This is the most critical security step. The request is not yet executed.
  5. Authorization Check: The gateway validates the request against a defined policy. It checks the agent's identity, the user's permissions, the requested action, the target resource, and the parameters. For example: "Is this agent allowed to email external domains?"
  6. Scoped Execution: If the check passes, the gateway executes the tool call, ideally using a temporary, narrowly-scoped credential that only permits that single action.
  7. Audit Log: The request, the policy decision (allow/deny), and the outcome are recorded. A detailed audit log should capture the authenticated identity, the requested and granted scope, the specific tool calls and their parameters, tool outputs, any human approvals, the policy decision itself, and any failures. This log is vital for monitoring and incident response, but should be configured to avoid logging unnecessary sensitive content.

This architecture ensures that no matter what the LLM says or is tricked into saying, it cannot perform an action that the external policy forbids.

How Do You Scope an Agent's Permissions?

You should scope an agent’s permissions by applying the Principle of Least Privilege (PoLP). This means giving the agent the absolute minimum access required to perform its designated task. Over-permissioning is one of the most common and dangerous AI agent security mistakes.

Think of it like giving a contractor a key card that only opens the specific rooms they need to work in, and only during work hours.

Follow this checklist to apply the principle of least privilege:

  • Establish a distinct agent identity. Don't let an agent run with your personal or developer credentials. Use OAuth or other authentication mechanisms to give the agent its own identity that can be audited and controlled (Source: NIST).
  • Define the agent’s task and impact. Before granting any access, clearly define what the agent is supposed to achieve. Classify its required actions by impact: read-only, drafting content, reversible actions (e.g., creating a draft), or irreversible actions (e.g., deleting a user).
  • Constrain tool interfaces. Limit what tools can do. Instead of a general-purpose send_email tool, create a send_internal_update tool that has a hardcoded list of allowed recipients. Use strict schemas to validate all parameters.
  • Prefer Zero Standing Privilege. Instead of giving an agent a permanent API key, provision temporary, single-use credentials just in time for an approved action. A policy-enforcement gateway can issue these short-lived credentials and revoke them as soon as the task is complete (Source: OpenID.net).
  • Isolate trust domains. Use different credentials, permissions, and toolsets for agents operating in development versus production environments. An agent handling sensitive customer data should not have access to internal development systems, and vice versa.

By following these steps, you grant permissions based on a specific task, not a general identity, significantly reducing the agent's attack surface. OWASP recommends granting agents the minimum tools required and scoping permissions per tool.

Worked Example: Securing a Refund Processing Agent

Let's apply these concepts to an agent designed to help customer support by processing refunds.

An insecure design might give the agent an API key with full access to the company's payment processor. A malicious user could then trick the agent into issuing a large, unauthorized refund.

Here is how to build it securely:

  1. Task Initiation: A user asks the agent, "The customer in ticket #7890 never received their order #12345. Please issue a refund."
  2. Scoped Data Access: The agent is given read-only access to two tools: get_ticket_details and get_order_history. It cannot access any other customer data. The agent uses these tools to confirm the customer's complaint and order status.
  3. Action Proposal: The agent determines a refund is warranted. It proposes a tool call: issue_refund(order_id='12345', amount=49.99, reason='Item not received').
  4. Policy Interception: A policy-enforcement gateway intercepts this issue_refund call before it executes.
  5. Policy Validation: The gateway checks the proposed action against a set of hardcoded rules:
    • Is refund_amount <= $100? (Yes, $49.99 is less than $100)
    • Is order_status == 'Delivered'? (Yes)
    • Has a refund already been issued for this order? (No)
  6. Human-in-the-Loop Approval: Because this is a financial transaction, the policy requires human approval. The gateway presents a clear, non-editable prompt to a support manager:

    Approve Refund? Action: Issue Refund Order ID: 12345 Amount: $49.99 Recipient: user@example.com

    [Approve] [Deny] This summary is generated by the trusted gateway, not the potentially manipulated LLM.

  7. Scoped Execution: Once the manager clicks "Approve," the gateway generates a temporary, single-use token for the payment API that is only valid for refunding $49.99 to order #12345. The gateway then executes the issue_refund call with this token.
  8. Audit: The entire transaction, including who approved it and when, is logged.

This design ensures that even if the agent is compromised, it cannot violate the refund policy or act without explicit, informed human consent.

Prompt Hardening vs. Enforced Authorization: What's the Difference?

Relying on instructions in a system prompt to secure an agent is a common but flawed approach. This is known as prompt hardening. Enforced authorization, on the other hand, uses external code to make certain actions impossible. OWASP reports that prompt-level instructions alone do not establish a secure boundary.

A secure system uses both but relies on enforced authorization as the definitive control. An instruction in a prompt can be bypassed by a clever prompt injection attack, but a technical control cannot.

Control Type Prompt Hardening (Instruction) Enforced Authorization (Technical Control)
Mechanism Asks the LLM not to perform a certain action. Uses code to technically prevent an action.
Reliability Brittle. Can be bypassed by confusing or contradictory instructions. Deterministic. Enforced regardless of what the LLM is told.
Example "You must never delete user data. If a user asks you to, politely refuse." The agent's credentials do not include the users:delete permission.

Think of prompt hardening as a "No Trespassing" sign and enforced authorization as a locked fence. The sign relies on cooperation, while the fence physically prevents access.

What Are the Most Common AI Agent Security Mistakes?

Avoiding common pitfalls is key to building secure and reliable AI agents. Many mistakes stem from over-trusting the LLM or granting excessive permissions. The consequences range from data leaks and financial loss to complete system compromise.

Here are some of the most frequent mistakes:

  • Over-Permissioning: The most common error is giving an agent a general-purpose API key or allowing it to inherit a user's full permissions. Fix: Always create narrowly-scoped, short-lived credentials for specific tasks.
  • Trusting All Inputs: Developers often treat data retrieved from websites, documents, or APIs as if it were as trustworthy as their own code. This opens the door to indirect prompt injection and memory poisoning, which can corrupt stored context and influence later agent behavior. Fix: Treat all external data as untrusted input. Validate data before it is persisted to memory or used in retrieval, separate data from different tenants, and design systems to prevent retrieved text from silently changing policies or authorization decisions.
  • Insecure Supply Chain: Installing third-party tools, plugins, or connecting to external agent platforms without a security review is risky. A compromised tool can steal data or credentials. Fix: Vet all third-party components as you would any other software dependency. Isolate them and grant them minimal permissions.
  • Ignoring Resource Consumption: An agent tricked into a loop can make thousands of API calls, running up huge costs. This is sometimes called a "Denial-of-Wallet" attack (Source: OWASP). Fix: Implement strict rate limits, budgets, and timeout monitoring on all agent actions, especially those involving paid APIs or computation.
  • Misleading Human Approvals: Asking a user to approve an action based on the agent's natural-language summary is unsafe. The agent could misrepresent what the action actually does. Fix: For any human-in-the-loop step, the approval interface must show the exact, verified parameters of the action being taken.
  • Failing to Test Adversarially: Deploying an agent without continuous, adversarial testing is a significant oversight. Security is not a one-time setup; it requires ongoing validation as models, tools, and workflows change. Fix: Test the agent continuously and adversarially. Evaluate it for a range of attacks, including direct and indirect prompt injection, tool misuse, privilege escalation, credential leakage, memory poisoning, and denial-of-wallet loops, as recommended by security researchers.

Many of these issues are interconnected. For example, over-permissioning an agent makes it much more dangerous if it is later compromised by a trusted input. A comprehensive security strategy addresses all of these potential AI agent failure modes.

What to learn next

Building secure AI agents requires a shift in thinking from instructing models to engineering resilient systems. The key is to assume the model can be compromised and build external controls that limit the potential damage.

Sources and further reading