01. AI Agent Platform Verification Guide
Before names and demos, verify sources, data boundaries, failure paths, and operational evidence.
An earlier version of this page described the features, integrations, pricing, and operational readiness of a product called Arrow 1.0 as facts without citing sources. Because it provided no evidence readers could independently verify—such as public documentation, repositories, or release notes—those claims have been removed. This page instead preserves a reusable verification process for evaluating AI agent platforms of the same kind.
This guide neither recommends a specific product nor guarantees that one exists. Treat vendor claims as unverified and make adoption decisions only within the scope supported by the evidence below.
1. Map the system boundary first
An "agent platform" is not a single model. It is generally an execution layer that connects user requests, model calls, tool execution, access to external data, state storage, observability, and approval workflows. Before focusing on the product name, draw the following data flow on one page.
- What inputs enter the system?
- Where are inputs and intermediate results stored?
- Which models and external services receive the data?
- Which credentials and permissions do tools use when they run?
- At which stages can a human approve or stop execution?
- What state remains after failure, retry, or partial success?
If the vendor documentation is insufficient to draw this flow, stop the evaluation. A feature list that says what the product "supports" does not explain where data moves or who is responsible for it.
2. Assign an evidence grade to every claim
Do not give every product claim equal weight. Evidence is more reproducible toward the top of this table.
| Grade | Evidence | What to verify |
|---|---|---|
| A | Public repository, version-pinned documentation, reproducible tests | Actual APIs, permissions, failure behavior, release history |
| B | Official security documentation, architecture documents, audit reports | Data retention, encryption, isolation, incident-response scope |
| C | A limited pilot you run directly | Latency, cost, quality, retries, operator intervention |
| D | Vendor demos, marketing, or verbal explanations | Record only as hypotheses to verify |
Broad phrases such as "production ready," "secure execution," or "supports every model" are not evidence on their own. Without supported versions, constraints, measurement periods, and failure cases, grade them D.
3. Verify security and permission boundaries
Agents can easily hold more authority than a simple chat UI. Connecting SaaS tokens, repository write access, customer data, and internal search systems in one execution graph turns a model's mistaken judgment into real side effects. Verify at least the following before a pilot.
- Are credentials separated from model context and logs?
- Does the platform support least privilege per tool and temporary credentials per execution?
- Does it inspect the boundary where external input can be elevated into tool instructions through prompt injection?
- Is explicit human approval required for high-risk actions such as writing, deleting, or sending?
- Are data and execution isolated across tenants, projects, and environments?
- Do audit logs record the requester, approver, model, tools, and policy versions used?
- Can the user control retention periods and deletion procedures?
The self-hosted deployment model does not solve these problems by itself. It merely transfers operational responsibility, updates, key management, and log-access control to the user's organization.
4. Falsify operational claims with a small pilot
Creating failure conditions reveals more than reproducing a demo. Choose one low-risk, read-only task and prepare a fixed input set with expected results. Include not only normal inputs but also empty responses, slow APIs, rate limits, duplicate events, incorrect tool results, model refusals, and restarts from an intermediate stage.
Measure all of the following during the pilot.
- Task success rate and the percentage corrected by a human
- Model and tool cost per request, plus worst-case latency
- Whether retries create duplicate writes or sends
- Whether execution can resume after interruption
- Whether logs alone reveal the cause and blast radius of a failure
- Whether regressions are detectable when model, prompt, or tool versions change
Do not base adoption on averages alone. Record the upper tail of expensive requests and long delays, the input types where failures cluster, and the boundaries that require human intervention separately.
5. Document the adoption decision
The final decision should contain verification results, not a product introduction.
- Allowed scope: Which data and tasks may the platform handle?
- Prohibited scope: Which high-risk tasks will not be automated?
- Evidence: Which documents, tests, and measurements support each key claim?
- Controls: Where are permissions, approvals, spending limits, and the kill switch?
- Exit plan: Can workflows, logs, and data be exported?
- Re-evaluation triggers: What must be verified again when the model, pricing, policy, vendor, or data flow changes?
When evidence is insufficient, "defer adoption" is a complete conclusion. Explicitly stating that a product cannot be verified is safer for readers and operational environments than inventing a plausible feature list for an unknown product.
Related documents
- AI:: index — evaluation principles for this section
- Technical paper review and presentation template — how to separate claims, evidence, and judgment
- Blog Editorial Guide — sourcing and fact-checking standards
References
- NIST AI Risk Management Framework — Govern, Map, Measure, and Manage activities for AI systems
- NIST AI RMF Playbook — concrete review activities organizations can apply