Tamper-Evident AI Agent Logs: Integrity, Gaps, and Limits

Delete the last record from a hash-chained log and the remaining records may still verify. Nothing inside that shortened file tells the verifier that another record once existed.

That is the first case I'd try when evaluating an agent logging system. Editing a middle record is the obvious demonstration. Removing a valid tail gets at a more interesting question: what does the verifier know from somewhere other than the file it's checking?

The chain needs a remembered endpoint

A chain links each record to the authentication value of its predecessor. Change a record and the link no longer agrees. Remove a middle record and continuity breaks. But the first two records of a valid three-record log are also a valid two-record history.

A retained checkpoint changes that. If a separate witness remembers that the stream reached record three with a particular chain head, the writer can't present two records as the whole history and still satisfy that checkpoint.

The witness has to retain information outside the writer's ability to replace it. Two files controlled by the same administrator don't create that separation. The interval between checkpoints also leaves a tail whose expected length the verifier may not know.

Run the failure cases

tamper_log.py makes those cases visible with standard-library Python:

python3 tamper_log.py

Each record contains a sequence number, boot epoch, event ID, action label, producer sequence, previous authentication value, and an HMAC. JSON key order and separators are fixed before the MAC is calculated.

The script writes three records and saves their count and chain head in a second temporary directory. That directory represents the witness in the experiment; both directories remain under the same local authority. The actions are labels, so a record named execution_receipt doesn't call an agent or payment service.

The HMAC key is a public demonstration string. HMAC is symmetric: someone who has the key to verify a record can also create one. This makes it easy to test signer compromise, but it doesn't provide a public-verification signature system or protected key storage.

The September 9, 2026 result file records 14 passing assertions from Python 3.13.12. Some related assertions are grouped below:

The restart test deserializes the saved records into fresh memory, verifies them, and appends. Crash recovery needs additional tests for interrupted writes, durable acknowledgements, and the order in which log and checkpoint state reach storage. Reloading valid JSON doesn't exercise those failures.

Integrity doesn't account for every event

The producer-gap case separates two sequences. The received producer numbers jump from one to three, while the collector stores two well-formed records. The collector's chain verifies. A check of the producer numbers sees the missing two.

If the producer never allocates the omitted event, neither sequence contains a gap. Detecting that loss requires another expectation: a downstream receipt, an acknowledged range, a heartbeat, or a known operation count. Signing the records that arrived cannot recover information that never reached the logger.

For a deployed system, assign event IDs before queueing and expose dropped-event counters through a separately monitored path. Decide how a full buffer affects consequential actions. Backpressure, blocking, and an explicit degraded state have different operating costs; silent loss leaves the reviewer unable to distinguish them later.

Multiple workers also need identifiable streams or a sequencing service. Wall-clock timestamps are useful context, but clock changes and concurrency make them insufficient for a reliable total order.

A valid receipt can still be false

In the last case, the signer appends a new record claiming execution after a valid checkpoint. The chain and old checkpoint both verify. The signer has preserved the past and lied about the next event.

To determine whether a refund happened, obtain a result from the payment service and link it to the action identifier. Check which identity could create that result and how duplicate delivery is handled. The agent's account of its own action and the downstream service's state answer different questions.

RFC 5848, Signed Syslog Messages, covers message authentication, ordering, replay resistance, and missing-message detection in its own protocol design. The small JSON/HMAC format here is separate, but the distinction between authenticated messages and independent knowledge of the stream is relevant to both.

Retain the reference and test recovery

A production witness needs its own access controls, stream identity, retention, and observable acknowledgement. Preserve enough information to prevent substituting one stream for another, and decide how long the writer may continue without a new checkpoint being accepted.

Storage retention helps preserve what was received. S3 Object Lock, for example, applies controls to object versions, with different bypass properties in governance and compliance modes. Permissions and retention settings matter as much as enabling the feature. It still cannot retain an event that never arrived.

For the next test, I'd stop the collector, continue a controlled workload, restart it, and reconcile the recovered log against downstream receipts. The interesting result is which actions and gaps can be accounted for after recovery. A verifier returning true for the surviving file is only one part of that answer.