45 Microseconds: The Cost of Governing Every Call

Every AI control layer has to sit somewhere. There are two real options: in the process making the call, or in a network gateway the call is routed through. The choice looks like an implementation detail and is not. It sets a latency floor you cannot engineer your way out of, and that floor decides how much of your traffic you can afford to govern.

The numbers

obsvr's SDKs run in-process. The full policy pipeline, meaning ruleset evaluation with normalization, the built-in PII scan, quota checks, customer hooks, multi-turn injection scoring, and shadow rules, adds 45.1 microseconds at the median in TypeScript, measured over 10,000 calls against an ungoverned baseline to the same mock. Signing alone, without any policy, is about 14 microseconds. On a 10 KB prompt the full pipeline rises to roughly 1,255 microseconds, because SHA-256 hashing and Unicode normalization scale with payload size.

A network gateway cannot get near that, and the reason has nothing to do with how well it is written. Before any policy runs, the call has to cross a network: TCP connection, TLS handshake or session resumption, request serialization, and the same again on the way back. Even inside one availability zone that is milliseconds. Across regions it is tens of milliseconds. The evaluation itself might take microseconds; the transport around it does not.

So the comparison is not two implementations of the same thing being faster or slower. It is a control that pays a round trip against one that does not, and the gap is roughly three orders of magnitude regardless of whose gateway it is.

Why the gap is structural

A gateway has to accept a TCP connection, terminate TLS, parse the request, evaluate, re-serialize, open a connection to the provider, and carry the response back the same way. Even with the evaluation itself taking microseconds, the transport does not. You can move the gateway closer, pool connections, and tune the stack, and you will still be paying for a network hop that an in-process check does not make at all.

This is not an argument that gateways are badly built. It is an argument that they are solving a different problem, and paying a fixed cost for it.

What the difference actually buys

Latency at this scale is not about the user waiting. Nobody notices tens of milliseconds on a call that already takes two seconds of model inference. It matters for a subtler reason: it determines what you are willing to put the control layer in front of.

An agent loop is not one call. A single agentic task can be dozens of model calls and tool invocations, and multi-agent setups multiply that again. At 50 ms per interception, a forty-step agent run pays two seconds of pure overhead. At 45 microseconds it pays under two milliseconds. The first number makes someone ask whether every step really needs screening. The second one never comes up.

The moment governance becomes something you apply selectively to keep latency down, you have a coverage gap, and coverage gaps are exactly what an audit finds. The cheapest control is the one you never have a reason to turn off.

What in-process gives up

Two real things, and neither is small.

Language coverage. A gateway governs anything that can make an HTTP request. An in-process SDK governs the languages it ships. obsvr ships TypeScript and Python; a Go service calling a provider directly is not covered by the SDK, and that is a genuine limit rather than a roadmap footnote.

It has to be installed. A gateway can be imposed at the network layer whether or not application teams cooperate. An SDK someone did not add governs nothing. This is the honest version of the "no code changes" claim every vendor in this space makes, ours included: auto-instrumentation means no changes to your call sites, not that the package installs itself.

These are the reasons obsvr also ships a gateway, and why the two together are the design rather than a hedge. The SDK sees intent, which the network cannot: which policy was in force, what the customer hook decided, which rule matched where. The gateway sees the wire, which the SDK cannot prove on its own: the raw response as the provider actually returned it. Neither one alone can demonstrate that nothing was routed around it. Together, with signed coverage heartbeats, they can.

The number to ask a vendor for

Ask for the p50 and the p99, on a stated payload size, against an ungoverned baseline, with the benchmark code published so you can re-run it. Ours is in bench/ in the SDK repository, along with two independent runs printed side by side and a disclosed note about where our own Python p50 measurements are bimodal at sub-150-microsecond scale. That note costs us something to publish. It is also the difference between a benchmark and a marketing number.

Reading a latency claim properly

A single median is close to meaningless on its own, and the reasons are worth knowing whoever you end up buying from.

Ask what is inside the number. Our 45.1 microseconds is the full pipeline. The stages are published separately so you can see where the cost lives: 13.6 microseconds for the base path of wrapping, event construction, the decision record, hashing and signing, then roughly 9 more for ruleset evaluation with Unicode normalization, another 9 for the built-in PII scan across 19 entity types, and the remainder for quotas, hooks, multi-turn injection scoring and shadow rules. A vendor quoting one number for "screening" is usually quoting the model call and not the surrounding work.

Ask what scales. Most of a governance pass is constant regardless of prompt size. Two things are not: SHA-256 content hashing and NFKC normalization both scale with payload. That is why our 45 microseconds becomes roughly 1,255 microseconds at a 10 KB prompt, and we publish the 10 KB rows rather than only the small-payload ones. If a benchmark only shows short prompts, it is showing you the flattering case.

Ask about the tail. p50 is the number that gets marketed and p99 is the number your on-call engineer experiences. Ours are in the same table: 49.7 microseconds at p95 and 72.5 at p99 for the full pipeline. Garbage collection and OS scheduling produce genuine outliers at high call volumes, and a benchmark with no visible tail has either not run long enough or has quietly trimmed it.

Ask what happens when the far end is slow. This is the question almost nobody asks and it matters more than the median. If the audit backend degrades, does the control layer block your call? We publish a run with the transport artificially slowed to 25 ms per POST: hot-path latency stays flat because delivery is fire-and-forget on a bounded queue, and the events that could not be delivered are counted as drops rather than silently lost. Both numbers are in the table, including the drop counts, because a system that stays fast by discarding evidence without saying so is not actually fast, it is just quiet.

The honest summary

In-process is roughly three orders of magnitude cheaper per call and cannot cover languages it does not ship or processes where nobody installed it. A gateway covers everything that speaks HTTP and pays a network round trip for every call it sees. Neither one is a strictly better design, and any vendor telling you their placement has no trade is describing marketing rather than architecture.

What is true is that the cost difference decides your default. At microseconds, governing every call is free enough that turning it off never comes up. At tens of milliseconds, somebody eventually proposes screening only the risky calls, and the moment that happens you have a coverage gap that an auditor will find before you do.