Choosing an AI Agent Memory Service: Security and Performance Tradeoffs

Deleting a support ticket is straightforward until an agent has turned it into a preference, a summary, and a few searchable fragments. The customer can lose access to the original ticket while the agent continues retrieving what it learned from it.

That's where I'd start a memory-service evaluation. I want to know who owns a memory, where it came from, and how to remove it before comparing retrieval speed. Those decisions become harder to change once several workflows depend on the stored material.

Decide what the agent is remembering

Session context, task state, user preferences, and source documents have different lifetimes. A shipping address supplied for one order shouldn't quietly become a permanent preference. A web page describing someone else's workflow shouldn't become an instruction for the next task.

Durable writes need an explicit policy. For each record, keep the tenant and subject, source identifier, writer, creation time, expiry, and trust state. If the record is a summary, retain the relationship to its sources. That relationship gives a deletion job somewhere to go after it removes the original document.

Trust state also needs an owner. A model's confidence in its summary isn't authority to promote retrieved text into policy. Preserve the source as data until a separate rule or authorized person allows that promotion. Signing the record can protect its provenance; it doesn't make the contents correct.

Carry the tenant through every operation

The service layer should take tenant scope from the authenticated request. Reads, writes, deletes, exports, cache keys, and background jobs all need the same binding. A missing tenant should fail the operation before it reaches storage.

Pinecone's namespace model is one documented example of tenant-scoped storage operations. A namespace gives the application somewhere to put a tenant's records. The application still has to choose the right namespace for the authenticated customer and protect the credential that can access it.

A good isolation test uses the same local record ID in two tenants. Read and delete it as tenant A, then check B's copy. Repeat the test through export and any background processing. This catches mistakes that a globally unique fixture ID can hide.

A local baseline you can inspect

memory_reference.py implements those checks in a small SQLite workload. It seeds four invented tenants with 256-byte text bodies, runs the lifecycle assertions, then times exact-key reads. Python's standard library is enough to run it:

python3 memory_reference.py

The retrieval function requires a known tenant, excludes records marked untrusted or expired, and returns the stored body with its source. The tenant argument represents scope already established by an application; authentication would happen before that call.

The tests cover seven areas: tenant-specific keys, missing or unknown tenant scope, trust filtering, expiry, deletion and export, preservation of another tenant's data, and rejection of a competing writer. Trust filtering here is a field check. Detecting malicious content before it gets marked trusted is a separate part of the application.

Recorded lookup timings

The September 9, 2026 results came from Python 3.13.12 and SQLite 3.50.4 on macOS/Darwin 25.6.0, arm64, with 18 logical CPUs. Each database used WAL mode on the local filesystem. Each reader had its own connection, 20 warmup reads, and 500 measured queries.

All 5,000 reads completed without errors. Latency covers the retrieval call and row fetch; percentiles use nearest rank. The throughput figure in the JSON includes thread setup and warmup, so it uses a different elapsed-time window. The lifecycle tests delete one row before timing, leaving 999 and 9,999 records.

The single-reader lookups were very short. Adding readers increased the observed latency, while increasing this small dataset by ten times made little difference. These short workstation runs are enough to inspect the method and local contention, but too small to estimate a production p99. They include no network, embedding, similarity search, reranking, or model generation.

After checkpointing, the database files occupied 348,160 and 3,387,392 bytes. The competing-write case deliberately uses a zero-second timeout while another connection holds a write transaction. It exercises SQLite's single-writer behavior in WAL mode, without measuring a sustained mixed read/write workload.

Move the same cases to a candidate service

I'd carry the isolation and lifecycle cases over before attempting a realistic retrieval benchmark. A successful delete should remove the record from queries and exports. Derived summaries and cached results need checks of their own. Backup retention is another question: removal from live retrieval doesn't establish physical erasure from every stored copy.

Then use representative tasks to measure the full retrieval path. Include concurrent updates, timeouts, overloaded queues, and deletion while a query is in progress. Record whether the returned material was relevant and permitted, along with the time it took to obtain it. A service that quickly returns another customer's record has failed before latency becomes interesting.

Export is worth testing early. Get the original text, identifiers, timestamps, provenance, and derivation links back out, then try rebuilding a small sample elsewhere. An export containing only vectors may leave you unable to reconstruct why a memory exists or which source deletion should remove it.

For cost, count ingestion, embeddings, storage, queries, reranking, retention, and export against that workload. I'd accept some added latency for a service whose ownership and deletion behavior I could explain. I wouldn't accept losing that control to save a few milliseconds on a lookup.