Prompt Injection in AI Systems

Fortify

Explanation

Prevent the assistant from seeing what the user should not see, and from acting on what an attacker wrote.

Insight

Filters that try to spot "malicious prompts" help, but they are probabilistic and attackers adapt. Design so that a successful injection still cannot reach anything valuable.

Practical

• Enforce access control at retrieval time. Store the source document's ACL with each chunk and filter by the user's identity on every query:

  def retrieve(query: str, user) -> list[Chunk]:
      groups = directory.groups_for(user.id)  # resolved per request, not cached at index time
      return vector_store.search(
          query,
          top_k=8,
          filter={"allowed_groups": {"$in": groups}},
      )
  

• Remove secrets from sources before indexing, then rotate them. Do not rely on the model to withhold them.

• Close exfiltration channels. Do not render markdown images or auto-fetch links from model output, or restrict them to an allow-listed domain.

• Mark untrusted content clearly in the prompt (delimiters, separate message roles), and tell the model it is data. This reduces, but does not prevent, injection.

• Add an input and output classifier for known jailbreak and injection patterns, such as Azure AI Content Safety Prompt Shields, Amazon Bedrock Guardrails or Meta's Llama Prompt Guard. Treat them as one layer.

• Protect agent configuration. Coding agents should need human approval to change their own settings, rules files or MCP configuration. Do not run them in auto-approve mode on untrusted repos.

Tools/Techniques

• Vector stores with metadata filtering (pgvector, OpenSearch, Bedrock Knowledge Bases metadata filters).
• Azure AI Content Safety Prompt Shields, Amazon Bedrock Guardrails, Llama Prompt Guard.

Metrics/Signal

• Indexed chunks without an ACL attached (target: zero).
• Assistants that render remote images from model output (target: zero).