LLMs and Agent Security: Why Controls Must Go Beyond the Model
Model safeguards are an important foundation. Organizations also need independent controls over the data, permissions, tools and actions surrounding AI systems.
A model can refuse an unsafe request and still be part of an unsafe application. Consider a support agent that can retrieve customer records and send email. An attacker hides instructions in a ticket telling the agent to forward those records to an outside address. Whether the attack succeeds depends on more than the model’s ability to recognize malicious text. It also depends on which records the agent can access, which destinations it can contact, and whether the sending action is independently authorized.
This is why security for LLMs and agents must extend beyond model behavior. Built-in safeguards reduce risk. The systems around the model must constrain what a mistake or successful attack can accomplish.
Key Takeaways
- Model safeguards reduce risk but cannot secure an application on their own, so AI agent security depends on controls over data access, permissions, tools, and actions around the model
- OpenAI, Anthropic, Google, and Microsoft all treat model behavior as insufficient on its own, which means critical boundaries must stay enforceable even when the model fails or is tricked
- Strong AI agent security needs five capabilities around the stack: runtime guardrails, tool and access control, agent discovery and inventory, kill switches, and red teaming of the full deployed workflow
- Before expanding an agent’s autonomy, teams should be able to show what it can access, how its actions are authorized, what adversarial testing found, and how its execution can be stopped
Provider Guidance Makes the Responsibility Clear
OpenAI’s safety guidance recommends moderation, adversarial testing, and human review of outputs. Its guidance for building agents adds input guardrails, structured data flows, and tool approvals, while acknowledging that agents can still make mistakes or be tricked. These recommendations call for safeguards in the application and workflow alongside protections supplied by the model.
Anthropic states the distinction directly in its Managed Agents security documentation: “model behavior is not a security control.” It recommends limiting tool access, restricting networking, and requiring approvals for risky actions. It also warns that its settings do not secure a customer’s custom tools, MCP servers, or self-hosted infrastructure. Those boundaries remain part of the deployment’s security responsibilities.
Google Cloud’s guidance for AI workloads on GKE says that even when a managed model has built-in safety filters, the customer still owns prompt-injection defenses for its specific application logic. It recommends inspecting prompts and responses and placing Model Armor between the application and inference endpoint. This is a concrete example of adding a security layer around model interactions.
Microsoft’s agent guidance recommends deterministic controls that block prohibited actions regardless of model output. It also calls for inventories of agents and their dependencies, least privilege, and reliable system-level mechanisms to pause or stop agents. Its shared responsibility model explains how ownership changes across deployment types while retaining customer accountability for data, authorization, and oversight.
Taken together, this guidance supports controls outside the model. It does not establish that every organization must buy a separate security product. Controls can come from a provider, an internal platform team, or a specialist. The architectural requirement is that critical boundaries remain enforceable even when model behavior fails.
Agents Make Action Security Essential
For an LLM that produces text, security includes what enters the context and what leaves the application. An agent adds an execution path. It can select tools, use credentials, retain memory, and initiate changes across systems. A problematic output can therefore become an operational incident before anyone reads a final answer.
A prompt saying “never send confidential information” communicates intent. An authorization check on an outbound tool call enforces a boundary. Both have a role, but they offer different assurances. Content classifiers and guard models can help identify suspicious behavior; action controls must also check the actual resource, operation, and destination.
Five Capabilities that Belong around the AI Stack
Runtime Guardrails
Inspect user inputs, retrieved content, tool responses, and model outputs for prompt injection, sensitive-data exposure, and policy violations. Put checks where data crosses trust boundaries, including paths that never reach the chat interface. Define what happens when a check detects a problem or becomes unavailable. A detection event is useful only if the workflow responds appropriately.
Tool and Access Control
Authorize actions independently of the model. Restrict the available tools and validate their parameters against the requesting user’s permissions and the agent’s task. For Model Context Protocol (MCP) connections, govern server access, credentials, and exposed operations. A permitted tool should still be unable to access an unauthorized record or send data to an unapproved destination.
Agent Discovery and Inventory
Maintain a current view of deployed agents, their owners, models, connected tools, and data access. Include agents created outside the central platform and delegated agents started by other workflows. Discovery makes it possible to identify where an approved model has been connected to an unapproved capability, or where permissions have expanded since the original review.
Kill Switches and Containment
Provide an administrative way to stop agent execution and prevent further actions without depending on the agent to obey a stop instruction. Depending on the architecture, containment may require canceling queued work, stopping delegated agents, disabling tool access, or revoking credentials. Test those mechanisms under realistic conditions. Stopping future execution does not reverse a completed payment or recover data already disclosed.
Model and Application Red Teaming
Test model behavior and the complete deployed workflow using red teaming. Include indirect prompt injection in documents and tool results, unauthorized data access, credential misuse, and attempts to bypass action approvals. Measure whether an attacker reaches the prohibited outcome, rather than only whether a detector flags suspicious text. Repeat testing when models, prompts, tools, permissions, or retrieval sources change. Provider testing cannot establish the security of every customer integration.
Consistent Policy Across Models and Frameworks
Organizations using several model providers need a consistent way to express and enforce their policies. A customer-data restriction should survive a model change. Approval requirements should remain effective when a team adopts a new agent framework. Discovery and incident response should cover the deployed system, including components outside the provider’s view.
Consistency does not require identical controls everywhere. A drafting assistant and an agent with production write access need different safeguards. Coverage should follow the actual data and execution paths, with explicit ownership of gaps and exceptions. External guardrails also require testing, careful permissions, and protection of the sensitive traffic they inspect.
Before expanding an agent’s autonomy, teams should be able to show what it can access, how actions are authorized, what adversarial testing found, and how execution can be stopped. The provider’s safeguards are part of that evidence. The controls surrounding the deployment complete it.
FAQs
A model can refuse an unsafe request and still sit inside an unsafe application. Whether an attack succeeds often depends on which records the agent can access, which destinations it can contact, and whether an action is independently authorized. Those boundaries live in the application and workflow, not in the model, so AI agent security has to extend past model behavior.
Traditional application security protects fairly predictable, deterministic software. An AI agent adds an execution path: it selects tools, uses credentials, retains memory, and starts changes across systems based on non-deterministic output. That means a problematic response can become an operational incident before anyone reads a final answer, so AI agent security has to authorize and constrain actions, not just inspect text.
Prompt injection is when an attacker hides instructions in content, an agent reads, such as a support ticket or a document, to make it act against intent. A content classifier can flag suspicious text, but it won’t always catch it. Pairing detection with an authorization check on the actual resource, operation, and destination is what reliably limits the damage.
Five capabilities belong around the AI stack: runtime guardrails that inspect inputs, retrieved content, and outputs; tool and access control that authorizes actions independently of the model; agent discovery and inventory; kill switches and containment to stop execution; and red teaming of the full deployed workflow. Together they keep critical boundaries enforceable even when model behavior fails.
