System Prompt Leakage
System prompt leakage refers to the risk that the system prompts or hidden instructions used to steer an LLM’s behavior may contain sensitive information that was not intended to be disclosed.
System prompts guide model behavior according to application requirements. However, they are not secure storage mechanisms and must not be treated as secrets or security controls.
Importantly, the real risk lies in what the system prompt contains and what the application improperly delegates to it.
Attackers can often infer guardrails and formatting rules simply by interacting with the model, even without directly extracting the exact wording of the system prompt.
Key Takeaways
- System prompt leakage occurs when the instructions used to steer an LLM's behavior are exposed to users, potentially revealing sensitive information like API keys, credentials, internal rules, or permission structures
- The system prompt itself should never be treated as a security control or used to store secrets; the real risk lies not in its disclosure, but in what it reveals about underlying security weaknesses and privileged access
- Even when the exact system prompt wording isn't disclosed, attackers can infer guardrails and behavioral restrictions simply by sending inputs and observing how the model responds over time
- Security controls such as privilege separation and authorization checks must never be delegated to the LLM via system prompts; these require deterministic, auditable enforcement in external systems
- Prevention requires keeping sensitive data entirely out of system prompts, implementing external guardrails independent of the LLM, and avoiding over-reliance on prompt instructions to enforce critical application behavior
Why System Prompts are Not Security Controls
System prompts can be influenced by prompt injection. They may be extracted through meta-prompt techniques. They are not deterministic enforcement mechanisms and should not contain sensitive operational data
If sensitive information (credentials, API keys, role definitions, connection strings) is embedded in system prompts, its exposure is a design failure, not merely a leakage event.
Additionally, if authorization rules or privilege logic are implemented inside the system prompt rather than in deterministic back-end systems, the architecture itself is insecure.
Common Risk Patterns
Exposure of Sensitive Functionality
System prompts may reveal API keys, database credentials, user tokens, system architecture details, tool configuration or back-end technologies. For example, If a prompt reveals the type of database being used, attackers may tailor injection attacks accordingly.
Exposure of Internal Rules
System prompts may describe internal thresholds or decision rules. For example:
- “Transaction limit is $5,000 per day.”
- “Total loan amount per user is $10,000.”
Attackers can use this knowledge to manipulate workflows, bypass controls or target logic weaknesses.
Revealing Filtering Criteria
A system prompt may instruct: “If a user requests information about another user, respond with, ‘Sorry, I cannot assist.’”
Knowing this rule allows attackers to craft bypass strategies.
Disclosure of Permissions and Role Structures
System prompts may reveal role definitions such as: “Admin users have full access to modify records.”
Attackers may then attempt privilege escalation.
Example Attack Scenarios
Scenario 1 – Credential Exposure
A system prompt contains credentials for accessing a tool. An attacker extracts the prompt and uses those credentials independently to access back-end systems.
Scenario 2 – Guardrail Bypass
A system prompt prohibits offensive content, external links and code execution. An attacker extracts these rules and then uses prompt injection techniques to override or bypass them, potentially enabling remote code execution.
The Core Insight
Disclosure of the system prompt is not the core vulnerability. The core vulnerability is storing sensitive data where it does not belong, delegating authorization logic to an LLM and relying on prompt text for enforcement of critical controls. Even if the exact wording of the system prompt remains hidden, attackers can often reverse-engineer guardrails through interaction.
Prevention and Mitigation Strategies
Separate Sensitive Data from System Prompts
Never embed API keys, authentication tokens, database names, role structures or permission mappings. Sensitive information must reside in secure backend systems inaccessible to the model.
Avoid Relying on System Prompts for Strict Behavior Control
LLMs are vulnerable to prompt injection. Critical controls (e.g., content filtering, policy enforcement) must be implemented outside the LLM in deterministic systems.
Implement External Guardrails
Use independent systems to inspect model outputs, validate compliance and enforce content restrictions. Model training alone is not sufficient.
Enforce Security Controls Independently
Authorization, privilege separation, and access control must occur outside the LLM. It must be deterministic and auditable and not rely on model reasoning. If agents require different privilege levels, use separate agents configured with least privilege.
Architectural Principle
Treat system prompts as configuration hints, not security boundaries. Security must be enforced at the application layer in back-end systems through deterministic access control mechanisms. LLMs are probabilistic systems. Authorization and security enforcement must not be probabilistic.
The Key Takeaway
System prompt leakage highlights a deeper issue. If leaking the system prompt breaks your security model, the architecture is already flawed. Do not store secrets in prompts. Do not rely on prompts for access control and do not treat hidden instructions as security mechanisms.
Security must exist outside the model.
< Back to Glossary of Terms