Choosing an AI Agent Memory Service: Security and Performance Tradeoffs

Agent memory is easy to describe as a retrieval feature and harder to operate as stored knowledge. Once an agent can reuse a statement next week, you need to know who supplied it, whose work it belongs to, when it expires, and how to remove it. Fast retrieval of the wrong customer's context is a security failure with a good latency number.

I would choose the memory model before choosing a service. Session context, durable user preferences, task state, and a searchable document collection have different lifetimes and trust assumptions. Putting them in one vector index doesn't remove those differences; it makes the application responsible for enforcing them.

Decide what may become durable

Keep ephemeral task context short-lived unless there is a reason to retain it. Durable memory needs an explicit write policy. A statement extracted from an untrusted web page should not become an authoritative user preference simply because the model summarized it confidently.

For each record, preserve the owning tenant, subject or workspace, source identifier, creation time, expiry, writer identity, and trust state. Where memory is derived from another record, retain that relationship. Otherwise deleting a source document can leave its conclusions circulating in summaries and embeddings with no obvious owner.

Treat stored instructions as untrusted content unless a separate authority approves their promotion. Provenance helps an agent or reviewer judge a statement; it does not establish that the statement is true. A signed record can faithfully preserve poisoned content.

Make tenant binding unavoidable

The application should derive tenant scope from authenticated context and pass it through every read, write, delete, export, cache lookup, and background job. A caller-provided tenant parameter is not authentication. Missing scope should reject the operation rather than select a global default.

For a documented vendor example, Pinecone describes serverless namespaces for tenant separation and tenant-scoped updates, queries, and deletion. Those are useful storage operations. The application must still bind the authenticated customer to the correct namespace and protect the service credential. I did not test Pinecone for this article.

Export matters as much as retrieval. Ask whether you can obtain original content, provenance, timestamps, identifiers, and the relationships used to derive memories. A vector-only export may preserve embeddings while losing the information needed to rebuild the system with a different representation.

A small reference experiment

The downloadable memory_reference.py uses only Python's standard library. It creates temporary SQLite databases, seeds four invented tenants, and measures exact-key reads of 256-byte text bodies at 1,000 and 10,000 initial records. Each concurrency setting uses 500 queries per reader, 20 unmeasured warmup reads per connection, and either one or four Python threads with separate connections.

python3 memory_reference.py

The run on September 9, 2026 used Python 3.13.12, SQLite 3.50.4, and macOS/Darwin 25.6.0 on arm64 with 18 logical CPUs. Databases used WAL mode on the local filesystem. The full JSON results include the method and storage sizes.

Percentiles use the nearest-rank method. Query latency covers the retrieval call and row fetch, excluding connection creation and warmup. The JSON's aggregate throughput includes thread setup and warmup, so its denominator differs. These are single short runs on an uncontrolled workstation, useful for reproducing the method and observing local contention. They are too small to estimate production tail latency or generalize a scaling curve.

All 5,000 timed reads completed without errors. Separate assertions checked tenant-specific keys, rejected absent or unknown tenant names, excluded untrusted and expired records, removed a deleted record from retrieval and export, preserved another tenant's record, and observed a locked competing writer. The scope parameter in this reference is a stand-in for authenticated context; the script implements no authentication service.

The database files occupied 348,160 and 3,387,392 bytes after checkpointing. No API charges were incurred; this is a local storage measurement, not an estimate of vendor cost. One row is deleted in each security test, so timed runs contain one fewer row than the initial count.

Read the limitations before the latency numbers

This workload has no embeddings, similarity search, network round trips, reranking, encryption service, or model generation. It measures indexed local lookups. Comparing its microsecond results to a hosted semantic-memory API would be misleading.

SQLite documents that WAL allows readers alongside a writer but only one writer at a time. The intentional lock test verifies a rejected competing write with a zero-second timeout. It does not measure mixed read/write capacity, recovery from a crash, or geographically distributed availability.

The poisoned-memory check is also narrow: a record marked untrusted is excluded. The experiment does not identify poison automatically or test a model's response to malicious instructions. That would require a separate task-level evaluation with provenance retained through retrieval and generation.

Evaluate the service lifecycle

For a real candidate, repeat the same ownership, expiry, deletion, and export cases through its supported API. Then measure representative search queries, document sizes, concurrent readers and writers, updates during retrieval, overload, timeouts, and the time until a successful delete is absent from results.

Separate logical deletion from physical erasure. Ask about replicas, backups, caches, derived summaries, retention holds, and the maximum completion time for each layer. Review encryption in transit and at rest, who controls keys, operator access, and how revocation affects existing copies. The local script's temporary-directory cleanup makes no secure-erasure claim.

Price the whole retrieval path: ingest and embedding, stored records, query volume, reranking, export, retention, and operational handling of failures. Use actual regional terms and measured request counts. A service with a convenient API may still make departure expensive if provenance cannot be exported.

My selection gate would be successful tenant and lifecycle checks before performance ranking. After that, compare task success and end-to-end latency on the same disclosed workload. The memory service should preserve the distinction between a fact, a preference, an untrusted claim, and a record that is no longer allowed to exist.