PII in AI Pipelines: Detect, Redact, or Block?

When obsvr's built-in scanner catches PII in a prompt or a tool call, it can do one of three things depending on how you've configured it: log it and continue, redact it and continue, or stop the call outright. Those are detect_only, redact, and block.

Picking between them isn't really an engineering decision. It's a risk and compliance call, and the config should just reflect whatever that call already is.

What actually gets detected

The built-in scanner runs a set of bounded regex detectors before the call goes out - no network round trip, nothing leaves your process to check. On the PII side that covers email addresses, phone numbers, SSNs, credit card numbers (Luhn-validated, to cut down on false positives), IP addresses, and UUIDs. On top of that it also catches a set of secrets - API keys across a few major vendor formats, AWS access keys, JWTs, private key blocks, GitHub tokens, Slack webhooks - and a set of prompt-injection patterns, since a lot of the same pre-call pass is a natural place to catch both.

What it doesn't catch with a regex: names, physical addresses, medical information, national ID numbers. Those need actual language understanding, not pattern matching, and require wiring up a Presidio instance alongside obsvr - the built-in scanner will tell you as much if you configure rules for those types without one connected.

Detection runs locally, before the request leaves your environment. The point is to stop transmission, not to analyze it after the fact.

detect_only: see what's actually flowing

In this mode, detected PII gets flagged in the audit record and the call goes out unchanged - the prompt reaches the model exactly as written.

This is the sane starting point if you're in a discovery phase and genuinely don't know what's moving through your pipelines yet. Even here, the audit trail is useful on its own: you end up with a record of every PII exposure event, which is a reasonable baseline for a GDPR data processing review.

redact: strip it before it leaves

Detected PII gets swapped for a typed placeholder before the call goes out - an email becomes [REDACTED_EMAIL], an SSN becomes [REDACTED_SSN], and so on for each type.

The model only ever sees the redacted version. For classification, summarization, and routing, that's usually fine - the structure and context survive redaction even when the actual values don't. The unredacted content never makes it into the audit ledger; the event records that redaction happened and which categories were involved, not the values themselves.

This is the right default for systems handling customer data where your LLM provider isn't covered under your DPA.

block: hard stop

Any detected PII rejects the call outright. A blocked_call event gets written, and nothing executes.

This makes sense where PII should never touch an LLM as a matter of policy - financial systems, healthcare environments with strict classification rules. It also works well as a developer safety net, catching accidental PII in a request before it ever reaches production.

The cost is friction: block mode will reject legitimate calls that happen to have PII mixed in with other data. Whether that tradeoff is worth it depends on whether upstream callers are already expected to have sanitized their input.

Choosing between them

A reasonable path: run detect_only for a couple of weeks to get a real baseline of what's flowing. If it's more than expected and your providers aren't covered by your DPA, move to redact. If your compliance posture needs a hard guarantee that PII never leaves your environment, use block.

Different pipelines in the same system can run different modes at the same time - a support pipeline on redact, an internal dev tool on block, a compliance reporting pipeline left on detect_only because the PII there is intentional and already governed elsewhere. The mode is a policy call; obsvr just enforces whichever one you picked and proves that it did.

Why regex is not the whole story, and where it stops

Detection splits into two classes, and conflating them is how teams end up with false confidence.

The first class is structured identifiers: card numbers, national IDs, API keys, email addresses, IBANs, phone numbers. These have shape. A card number has a length and satisfies a checksum. A cloud access key has a fixed prefix. Patterns handle these well, run in microseconds, and produce a deterministic answer reproducible months later from the pattern alone. Where a checksum exists, using it collapses the false-positive rate: a sixteen-digit number failing Luhn is almost certainly not a card.

The second class is contextual identifiers: names, addresses, locations, medical terms, anything where the same string is sensitive in one sentence and meaningless in another. "Ridge" is a surname, a street, and a geological feature. No pattern separates them, because the distinction lives in surrounding context rather than in the characters. This class needs named-entity recognition, which means a model, which means latency and a dependency.

obsvr splits along exactly that line. The built-in scanner covers 19 PII types: the structured ones deterministically in the hot path, and the contextual ones through an optional Presidio integration running as a local sidecar. The documentation states the limit plainly rather than in small print, because a scanner claiming to catch names by pattern is either wrong or flagging so much that nobody leaves it enabled.

The failure modes nobody plans for

Three things break PII detection in production, and none of them are the detector being bad at its job.

Obfuscation. The same identifier written with full-width digits, zero-width spaces between characters, or homoglyph substitutions defeats a naive pattern while remaining perfectly readable to the model. This is why normalization has to happen before scanning: Unicode normalization, invisible-character stripping, confusable folding, and decoding of percent, hex and base64 segments, so the scanner sees canonicalized text rather than wire format. Scanning the raw string alone is scanning the wrong thing.

The redaction that silently did not happen. Worse than a miss, because it produces false confidence. If the redaction step fails after the detector fired, and the system forwards the original content while the audit record says "redacted," you have both the leak and a record asserting the opposite. A partially-redacted request that reads downstream as a clean one is the worst available outcome. The correct posture is to block rather than forward content that could not be guaranteed redacted, and to make the record say what actually happened.

The response side. Most teams scan prompts and forget the model's output goes into logs, downstream tools, and other agents' context. A support agent summarizing a ticket thread can reproduce a customer's email address in its summary. Scanning one direction is half a control.

What the record has to contain

Detection is only half the obligation. The other half is demonstrating later that it ran.

An auditor asking whether customer data ever reached the model provider does not want a dashboard count. They want a specific call, showing which types were detected, what action policy took, and which ruleset was in force at that moment. That means the event carries the detection result and the policy version, and the redaction marker in the stored copy distinguishes "we scanned and removed something" from "we could not scan this." Those are different facts, and an auditor who cannot tell them apart has to treat every redaction as possibly the second one.

This is where detect_only earns its place beyond being a soft launch. Running it for a week produces the baseline for the argument you will eventually make: here is what actually flows through our pipelines, here is what we chose to enforce, and here is the record showing enforcement ran on every call rather than the ones we happened to sample.