What AI Agent Assurance Claims Actually Cover

An agent supplier hands you a certificate. The sales team says it covers AI security. Your application will let the agent retrieve customer attachments and request payments. Where, exactly, were those permissions tested?

That's the point where I want to leave the summary slide and read the report. The certificate may be legitimate and relevant while covering a different part of the system from the one I'm worried about.

Read the scope before the result

ISO/IEC 42001:2023 sets requirements for an organization's AI management system. That can tell a buyer something useful about how AI work is governed. To judge attachment access, though, the buyer also needs to know how the application binds a request to a customer and where it rejects access to another customer's records.

A SOC 2 report raises a related question: which service and controls did the examination cover? A Type 1 report concerns a specified date; Type 2 includes operating effectiveness over a period. The AICPA report review checklist is a useful guide to reading the system description, exceptions, reporting period, and responsibilities left with the customer.

A technical assessment can examine the application more directly. Its relevance still depends on the tested configuration. Results from a read-only assistant don't carry over automatically when the same assistant gains a payment tool or a service account with wider access.

I would write down the permission I'm relying on and look for the corresponding test. If I can't find it, I'd ask for that test rather than trying to infer the answer from the report's title.

Different programs answer different questions

These are examples of published program scopes. I am affiliated with TKOResearch, which offers AgentBoundary; its inclusion here is a description of our program, not an independent provider ranking.

Organizational governance and technical testing can both matter to the purchase. Governance affects who approves a new connector and how failures are handled. Technical tests show what happened when a particular configuration encountered a particular case. I want those two accounts to connect: the release process should know when a change makes an earlier test result less relevant.

Follow a payment through the test

Take a purchasing agent that can draft an order but needs a separate approver before payment. Give it a retrieved document containing an instruction to approve its own order. The interesting result is what reaches the payment service.

If the model refuses, record that behavior. Then exercise the payment boundary directly with a request lacking valid approval. The API should enforce the payment rule even if a later model produces a different answer. The test record needs the initiating user, executing service identity, order, approval, and downstream result so the reviewer can see where the request stopped.

I'd then change one thing at a time. Keep the approval but alter the amount. Use another customer's order. Let the approval expire. Retry an operation that already completed. Those cases expose whether the system treats approval as permission for a specific operation or merely remembers that someone clicked a button.

Successful cases belong beside the denials. An authorized draft should be created, and a correctly approved payment should complete once. Otherwise the tests could all pass because the integration is broken or permanently blocks useful work.

A single percentage conceals too much here. I'd rather read five cases with clear outcomes than a combined score that mixes an unauthorized payment with a harmless formatting mistake.

Keep the report attached to the release

The tested model is only one part of the configuration. Prompts, retrieval sources, memory, tools, service credentials, and downstream policy all affect what an agent can do. The AgentBoundary scope rules describe changes that require that relationship to be reconsidered.

Keep the report reference with the release configuration and name someone responsible for deciding what needs to run again. A new attachment connector might need new ownership tests even when the model hasn't changed. A broader service role can invalidate an access assumption without changing a line of prompt text.

For the purchasing agent, the useful assurance is specific: this build, with these permissions, rejected these unauthorized payments and completed these approved ones. When one of those conditions changes, the team has a test to repeat and a reason to repeat it.