AI Assistants: The Hidden Security Gap and How to Close It

In the rapidly evolving world of AI, cutting-edge tools are transforming how we work. However, a recent discovery highlights a critical security blind spot that could expose sensitive data: retrieval-permission boundaries in AI assistants. Egiziago Cioffi, an IT and Enterprise Architect, stumbled upon this issue while building an Azure OpenAI email assistant. Despite passing all standard evaluations, the assistant was inadvertently providing information to low-privilege users that they shouldn't have had access to.

The Silent Security Flaw: Permissions Mismatch

Cioffi's investigation revealed that his Azure OpenAI assistant, when handling customer emails, was resolving about 60% of inquiries. The problem emerged when he tested the assistant with a low-privilege account. The assistant returned SharePoint content that this user couldn't access directly through SharePoint. This pointed to a fundamental flaw: the AI was operating with the permissions of the indexer, not the requester.

This isn't an isolated incident. Research from Straiker's red team found that 91% of successful attacks on production agents resulted in silent data exfiltration, meaning sensitive information was leaked without detection. While these attacks could stem from various vulnerabilities, the core issue often lies in how AI assistants access and retrieve data.

The U.K.'s AI Security Institute also documented instances of AI agents acting outside their intended scope during testing. These findings underscore a broader challenge: current evaluation methods often focus on the accuracy and relevance of AI responses, but they overlook a crucial aspect – whose permissions are being used to fetch the underlying data.

Why Standard Evaluations Miss the Mark

Cioffi's team ran rigorous evaluations designed to test factual accuracy, relevance, and task completion. However, the question of whether the retrieval pipeline respected user-specific permissions was never part of the framework. This oversight is common because:

• Focus on Output, Not Access: Evaluations primarily measure the quality of the AI's final answer.
• Complex Pipelines: Custom retrieval pipelines can bypass native security features, making it harder to track data access.
• Service Account Permissions: AI agents often run under a single, powerful service account, masking individual user access levels.

Adriel Desautels, CEO of Netragard, explains this as a "structural collapse of authorization boundaries." If an AI's underlying service account has broad permissions, and the application doesn't enforce identity-aware retrieval, users with lower privileges can inadvertently access restricted data. This effectively lowers the security bar to the lowest common denominator.

Cioffi's Solution: A Query-Path Filter

Fortunately, closing this security gap doesn't necessarily require a complete overhaul or a new identity platform. Cioffi's solution involved implementing a query-path filter directly into the retrieval process. This filter checks the requesting user's permissions against the content *before* it's fed into the AI model's context window. If the user lacks the necessary permissions in SharePoint, that content is simply not retrieved.

The trade-off is a narrowed retrieval scope. The assistant may now exclude content that it could previously access, potentially impacting its ability to answer certain queries. However, for sensitive data environments, this is a necessary compromise to maintain security and enforce proper authorization boundaries. Cioffi's assistant still manages to auto-resolve a significant portion of emails, demonstrating that a balance can be struck.

Beyond Identity Governance: The Retrieval Boundary Layer

While major security firms like CrowdStrike and Palo Alto Networks are heavily investing in identity governance platforms (focusing on service account lifecycle, permissions, and token expiry), these solutions operate at a different layer. They manage the credentials themselves but don't inherently govern the retrieval-time permissions for AI agents interacting with data on behalf of users.

The critical layer that needs attention is the retrieval permission boundary. This is where an AI assistant, even when using a properly authenticated service account, must respect the access rights of the end-user making the request. Azure AI Search's native ACL trimming is an example of a control at this layer, but it's not universally applied, especially in custom pipelines.

A Simple Test for Enhanced Security

To identify and address this vulnerability, security teams can implement a straightforward test:

• Ask the Right Questions: Inquire about the permissions model for your AI retrieval system.
• Verify Native Controls: If using Azure AI Search with SharePoint and Entra principals, confirm that query-time ACL trimming is enabled.
• The Two-Account Test: Run the same query from both a high-privilege and a low-privilege account. Compare the results. If the low-privilege account receives more information than it could access directly, the retrieval permission boundary is not enforced.

This simple, thirty-minute test can reveal critical security gaps that standard evaluations miss, helping organizations ensure their AI assistants are not inadvertently becoming gateways to sensitive information.