How to Evaluate Tools for Securing Production AI Agents

I would give an agent-security vendor a workflow before giving it a questionnaire. A hypothetical support agent can read a ticket, retrieve an attachment, draft a reply, and request a refund. Each step has a different security decision. A tool that screens the incoming prompt may have no control over the credential used to retrieve the attachment or the payment API that executes the refund.

Start by drawing those decisions and naming the component that can enforce each one. Then compare products at the points where they actually operate. This method doesn't produce a universal best-tool list. It produces a defensible choice for an agent with known access and a known operating environment.

The standards and protocol references here were checked on September 9, 2026. The test cases below are an evaluation plan, not results from commercial products. No vendor runtime was tested for this article.

Locate the control before comparing features

Divide the workflow into input handling, model execution, tool dispatch, downstream resource access, and result handling. Include every route that could bypass the tool gateway: SDK calls, shell access, browser sessions, background workers, and retry queues.

A content filter might flag instructions in an attachment. An authorization service might reject a refund outside the agent's authority. A sandbox might constrain filesystem access. An egress gateway might restrict destinations. These controls solve different problems, so a single detection score cannot establish the adequacy of the whole path.

NIST SP 800-207 is useful background for separating policy decisions from enforcement and concentrating on access to resources. Applying that architecture to agents is an engineering interpretation; the publication does not certify an agent gateway or prescribe a particular AI-security product.

Write acceptance cases around the action

Use a disposable test application with invented customers and inert actions. Make the service return a receipt for accepted operations so the evaluator can distinguish a proposed call from a completed action. A model transcript alone is insufficient.

The canary is a non-secret string in the lab, not a production token. The destinations and tool receipts should stay inside the test environment. Security evaluation doesn't require giving the candidate a live payment capability.

For tool-connected systems using HTTP, the MCP authorization specification dated 2025-11-25 describes requirements including token audience validation and resource indicators. Pin the protocol revision when reviewing a supplier's claim. Protocol compatibility alone still doesn't answer whether the downstream service enforces customer ownership of an attachment.

Treat approvals as data with a lifetime

A human approval needs a precise object: actor, tenant, action, resource, arguments, expiry, and allowed execution count. Evaluate what happens if the model changes the recipient after approval, a queue retries an old request, or the approving person loses their role before execution.

For the refund example, I would expect the approved customer, transaction, amount, and currency to remain bound to the execution request. If the tool only checks that an approval button was clicked somewhere in the conversation, the system has lost the relationship between permission and action.

The same reasoning applies to secrets. Ask where credentials are minted, how narrowly they are scoped, which component can read them, how they expire, and how revocation reaches a running worker. A redacted UI cannot establish that a credential never enters a model prompt or a debug log. Inspect the applicable path in the test environment.

Measure the operational cost of the control

Time the workflow with and without the candidate under the same inputs, model settings, and downstream service behavior. Record p50, p95, and p99 added latency, successful authorized operations, denied operations, errors, and timeouts. Separate cold starts from warm requests and include concurrent work from a noisy synthetic tenant.

Retries require their own result category. A tool may look fast on its successful requests while failing enough calls to increase end-to-end completion time. Report the total elapsed time and final outcome for each user task as well as per-call latency. Set the acceptance budget before seeing the vendor's graph.

Don't casually decide that every dependency failure should block every operation. Stopping a read-only documentation assistant has a different cost from allowing an unapproved refund. Define which operations can continue with bounded cached policy, which must wait, and how the operator sees degraded coverage. Then force those conditions in the lab.

Keep documented claims and measured behavior in separate columns

For each candidate, retain the product edition, version or hosted-service evaluation date, deployment mode, configuration, supporting documentation, and observed result. Mark unavailable cases as untested. A salesperson's answer can help locate documentation; it should not become a passed test.

Use hard gates for failures your organization cannot accept, such as cross-tenant reads or approval bypass. Only compare usability, latency, deployment effort, and cost among candidates that meet those gates. A weighted average can hide a fatal boundary failure under excellent scores for easy features.

I would finish the selection with one replayable package: the workflow, inert fixtures, expected outcomes, recorded receipts, failure cases, and a short list of deployment assumptions. Re-run the relevant parts when tools, permissions, model configuration, or network paths change. That keeps the purchasing decision connected to the system people will actually operate.