64 Microseconds: The Cost of In-Process Governance

Every AI control layer has to sit somewhere. There are two real options: in the process making the call, or in a network gateway the call is routed through. The choice looks like an implementation detail and is not. It sets a latency floor you cannot engineer your way out of, and that floor decides how much of your traffic you can afford to govern.

The numbers

In the published 0.11.2 benchmark, Obsvr's TypeScript full-local pipeline measured 63.2–64.1 microseconds p50 across two runs. The base signed/enqueued path measured 24.7–25.1 microseconds, and the 10 KB full-local row measured 1,749.4–1,756.3 microseconds. Python's small full-local row measured 524.3–525.1 microseconds. These are local mock-transport deltas, not provider latency.

A network gateway cannot get near that, and the reason has nothing to do with how well it is written. Before any policy runs, the call has to cross a network: TCP connection, TLS handshake or session resumption, request serialization, and the same again on the way back. Even inside one availability zone that is milliseconds. Across regions it is tens of milliseconds. The evaluation itself might take microseconds; the transport around it does not.

So the comparison is not two implementations of the same thing being faster or slower. It is a control that pays a round trip against one that does not, and the gap is roughly three orders of magnitude regardless of whose gateway it is.

Why the gap is structural

A gateway has to accept a TCP connection, terminate TLS, parse the request, evaluate, re-serialize, open a connection to the provider, and carry the response back the same way. Even with the evaluation itself taking microseconds, the transport does not. You can move the gateway closer, pool connections, and tune the stack, and you will still be paying for a network hop that an in-process check does not make at all.

This is not an argument that gateways are badly built. It is an argument that they are solving a different problem, and paying a fixed cost for it.

What the difference actually buys

Latency at this scale is not about the user waiting. Nobody notices tens of milliseconds on a call that already takes two seconds of model inference. It matters for a subtler reason: it determines what you are willing to put the control layer in front of.

An agent loop is not one call. A single task can involve dozens of model calls and tool invocations. At 50 ms per interception, forty steps pay two seconds of overhead; at roughly 64 microseconds, the same arithmetic is about 2.6 milliseconds on the measured TypeScript full-local path.

The moment governance becomes something you apply selectively to keep latency down, you have a coverage gap, and coverage gaps are exactly what an audit finds. The cheapest control is the one you never have a reason to turn off.

What in-process gives up

Two real things, and neither is small.

Language coverage. A gateway governs anything that can make an HTTP request. An in-process SDK governs the languages it ships. obsvr ships TypeScript and Python; a Go service calling a provider directly is not covered by the SDK, and that is a genuine limit rather than a roadmap footnote.

It has to be installed. A gateway can be imposed at the network layer whether or not application teams cooperate. An SDK someone did not add governs nothing. This is the honest version of the "no code changes" claim every vendor in this space makes, ours included: auto-instrumentation means no changes to your call sites, not that the package installs itself.

These are the reasons Obsvr also ships a gateway. The SDK sees application intent; the gateway sees supported provider wire traffic and emits signed coverage heartbeats. Combining them creates an independent reconciliation signal, while network egress policy and deployment inventory are still required to claim complete routing coverage.

The number to ask a vendor for

Ask for the p50 and the p99, on a stated payload size, against an ungoverned baseline, with the benchmark code published so you can re-run it. Ours is in bench/ in the SDK repository, along with two independent runs printed side by side and a disclosed note about where our own Python p50 measurements are bimodal at sub-150-microsecond scale. That note costs us something to publish. It is also the difference between a benchmark and a marketing number.

Reading a latency claim properly

A single median is close to meaningless on its own, and the reasons are worth knowing whoever you end up buying from.

Ask what is inside the number. The published table separates the base signed/enqueued path, rules, PII scanning, quotas, and the full-local policy path. That makes the 63.2–64.1 microsecond TypeScript result reproducible rather than collapsing unlike work into one screening number.

Ask what scales. Hashing, normalization, and scanning scale with payload. The measured TypeScript full-local p50 rises from 63.2–64.1 microseconds on the small payload to 1,749.4–1,756.3 microseconds at 10 KB, which is why both rows are published.

Ask about the tail. The current report publishes p95 alongside p50 and retains both complete runs. It also discloses Python's wider scheduling variance rather than trimming the slower pass.

Ask what happens when the far end is slow. This is the question almost nobody asks and it matters more than the median. If the audit backend degrades, does the control layer block your call? We publish a run with the transport artificially slowed to 25 ms per POST: hot-path latency stays flat because delivery is fire-and-forget on a bounded queue, and the events that could not be delivered are counted as drops rather than silently lost. Both numbers are in the table, including the drop counts, because a system that stays fast by discarding evidence without saying so is not actually fast, it is just quiet.

The honest summary

In-process governance avoids a network round trip but cannot cover languages it does not ship or processes where nobody installed it. A gateway can cover supported HTTP provider traffic routed through it and pays a network round trip. Neither placement is universally better.

What is true is that placement changes the cost structure. Low in-process overhead makes broad adoption practical, but complete coverage still depends on routing through a documented enforcing boundary and accounting for any sampled or lost evidence.