EngineeringJune 9, 20266 min read

How obsvr's Tamper-Evident Chain Works

Every audit event is signed on the client, chained to the one before it, and countersigned by the server. Here's the exact mechanism, and why a client-side signature alone isn't enough.

When an auditor asks "can you prove this log wasn't modified," most teams open a database table and hope the timestamp looks convincing. The row is there, but there's no actual mechanism proving nobody edited it after the fact.

obsvr's event chain is built to make that question answerable with something closer to mathematical certainty, and it takes two separate signatures to get there - not one.

The client-side chain

Every event carries a monotonically increasing seq_no, an sdk_session_id that stays stable for the life of the SDK process, an sdk_sig - an HMAC-SHA256 signature over that event's canonical payload - and a prev_sig pointing at the signature of the event before it.

That last field is what turns individual signed events into a chain. Editing any event invalidates every signature computed after it, so tampering isn't silent - it breaks something downstream that anyone verifying the chain would notice.

Why the client signature isn't the whole story

Here's the part that's easy to gloss over: the SDK's signing key is derived from your API key. That's convenient, and it's enough to prove the chain hasn't been tampered with in transit or at rest by anyone who doesn't have the key. But it means anyone who does have valid API key access - including, worst case, an attacker who's compromised the client - could in principle recompute that same key and produce a signature that looks legitimate.

A client-only HMAC chain proves internal consistency. It doesn't prove the client itself wasn't the thing lying to you.

The server countersignature

That's the actual reason obsvr's ingest service countersigns every event it accepts, using a key that's generated and held server-side and never touches the SDK or leaves obsvr's infrastructure. The countersignature is versioned, so key rotation doesn't invalidate historical records - older events just verify against the key version they were signed with.

A record only counts as fully verified once both signatures check out: the client chain proving nothing was edited after capture, and the server countersignature proving the event genuinely passed through obsvr's ingest and wasn't fabricated wholesale by a compromised client. Ingest also rejects duplicate and replayed events at write time, so the same signed event can't be resubmitted to pad a record.

Verifying a chain

Verification walks the chain from the first event forward: recompute the HMAC for each event's canonical payload, confirm it matches the stored sdk_sig, confirm the countersignature matches the key version it claims, confirm the next event's prev_sig points where it should, and repeat to the end. If any event was altered, even by a single character, the recomputed signature won't match and you know exactly where the chain broke.

Why this matters for compliance

SOC 2 Type II asks for evidence of continuous monitoring and sustained control effectiveness. A log is only useful as evidence if you can show it reflects what actually happened, not what someone decided afterward it should say.

EU AI Act Article 12 asks for logs "robust against manipulation." A plain database table doesn't meet that bar - anyone with write access can edit a row. A dual-signed chain meets it by construction, not by policy, and specifically closes the gap a client-only signature leaves open.

The Merkle layer, for good measure

For larger deployments, obsvr also runs a daily job that folds the previous day's events into a Merkle root, which gets anchored to an external, immutable record. Even in the unlikely event that both signing keys were somehow compromised at once, rewriting anything older than a day would conflict with a public commitment that can't be quietly altered.

Export and independent verification

The full chain exports as NDJSON, one event per line in order, with both signatures and enough metadata for any third party to verify it independently - no access to obsvr's systems required, and no need to trust obsvr's word for it. That's what "your auditor can open a ZIP and check it themselves" actually means in practice.

What the chain proves, and what it does not

This is the part most descriptions of tamper-evident logging skip, and skipping it is how people end up trusting the wrong property.

The chain proves integrity and ordering. If an event was altered after signing, verification fails. If an event was removed from the middle, the linkage breaks at that point. If events were reordered, the sequence check catches it. Those are strong properties and they hold without anyone having to trust obsvr, because the verifier recomputes the signatures itself.

The chain does not prove that the recorded content is true. If the SDK is fed a fabricated prompt, it will faithfully sign the fabrication. Cryptography attests that a record has not changed since it was written; it says nothing about whether the thing written was accurate. Anyone claiming otherwise is overselling what a signature is.

It also does not, on its own, prove completeness. A chain of a hundred events is internally consistent whether or not there were a thousand calls. This is why the sequence numbering and the coverage heartbeats exist: they turn "we have no record of that" into a checkable statement rather than an absence someone has to interpret. A missing sequence number is a positive fact. Silence is not.

Being precise about this matters commercially, not just intellectually. A security reviewer who finds you overstating a cryptographic property will discount everything else you claim, and they should.

Why HMAC and not public-key signatures on the client

A reasonable question from anyone who has worked with signing systems: why is the client-side chain an HMAC, which requires a shared secret, rather than an asymmetric signature anyone could verify with a public key?

Because of where the key would have to live. An asymmetric private key sitting inside a customer process is a private key on every machine running the SDK, and that key becomes the thing an attacker wants most. The client chain is derived from the API key the customer already holds, which means the SDK introduces no new long-lived secret to protect, and the trust properties are the ones the customer already accepted when they provisioned that key.

The asymmetric signature exists where it can be protected properly: the server countersignature, and the Ed25519 signature over the daily Merkle root. That is the layer an external party verifies against a published public key, and it never requires access to anything customer-side.

The split is deliberate. The client chain proves the sequence held together in the customer's process. The server signature proves obsvr accepted the event and cannot later deny it. The Merkle root proves the day's set existed in that exact form at that time. Each layer covers a different adversary, and no single one of them is the whole argument.

Running the verification yourself

The property that matters most is that none of this requires taking our word for anything. The verifier ships with the SDK in both languages, and the CLI runs in two modes: a structural check with no key at all, which validates chain linkage, sequence continuity, session consistency and timestamp monotonicity, and a full check with the API key, which recomputes every HMAC.

The structural mode is the interesting one for an auditor, because it lets someone verify the shape of the record without ever holding a credential that could produce records. They can confirm nothing was removed or reordered without being handed the ability to forge an entry.

If you are evaluating any system that claims a tamper-evident trail, that is the test worth running: ask whether a third party can verify it without a secret, and without calling the vendor's API. If verification requires an API call to the vendor, the vendor is inside the trust boundary of their own evidence, and the property you were sold is weaker than it sounded.