Test policies against real conversations

LLM chats created a new route for sensitive data to leave the company. Rules based on people and apps couldn’t explain what would happen to the content of a message.

Try a message. Check the rule. Try again.

Try the same message against two example policies. The app rule blocks access; the content policy checks what the message contains.

Illustrative comparison · Sales team using ChatGPT

Rule-based policy

Describes conditions

  1. IfGroup is Sales
  2. AndApp is ChatGPT
  3. ThenBlock

App blocked · Sales + ChatGPT

Prompt-based policy

Looks at the message content

Example prompt

Hi, if you need to follow up, email me at jane.doe@acme.com

Anonymized due to Data Privacy

A safe place to test policies

I proposed a chat-style simulation panel beside the policy editor. Admins could test real messages, inspect what was blocked or anonymized, then refine the matching rule, all before applying the policy to employee conversations.

Choosing the side panel

I compared a quicker-to-build modal with a chat-style side panel. The chat pattern matched how people send data to an LLM, making policy testing feel familiar.

Hover or select an option to compare

Why the panel

People share data with LLMs through chat. I used the same pattern so admins could test policies through the conversations those policies would protect.

I compared both layouts with around 10–12 colleagues across four policy tasks. The internal evaluation favored the side panel.

From a result to its rule

Test a message, inspect what was blocked or anonymized, then open the matching rule. The loop shows how an admin moves from a result to the setting behind it.

Start with a real message.
12 SEC / INTERACTION LOOP
The changed message, its matching category, and the route to the rule stay together.

A clearer wait

Tests could take up to 30 seconds. I recognized an opportunity to make that wait educational: show when policies were updating and when the message was being tested, to make the process clearer and build trust in the result.

▷   Policy Simulation

Updating policies

Loading your latest policy changes so the test uses the current rules.

Hi, if you need to follow up with me, email jane.doe@example.com.

Updating policies
The tester loads the latest policy, then checks the message. Existing prototype animation; the sequence below is shortened for this presentation.

Stronger policies across more use cases

After launch, security analysts reported improvements in their day-to-day policy work:

Fewer policy issues
Testing real messages helped analysts identify and resolve problems in their policies.
New protection gaps uncovered
Simulation surfaced overlooked data sources and scenarios their existing rules did not cover.
Broader policy coverage
Analysts used those findings to refine policies for a wider range of real-world use cases.
Qualitative feedback following production release.