Can You Trust an AI Data Agent? A Six-Step Decision Checklist
AI data agents can turn a question into an analysis, dashboard, or proposed action in minutes. That changes the speed of the work, not the burden of proof.
In its 10 September 2026 announcement, OpenAI says the Data agent in ChatGPT Work can use approved sources, business terms and metric definitions, enforce the connected account's permissions, expose evidence behind findings, and carry out connected-tool actions that a user approves. Those are useful controls. They are not, by themselves, proof that a result is correct or safe for your organisation.
Here is the checklist I would use before relying on any data agent.
1\. Make it reproduce a trusted metric
Choose a closed historical period and a metric that finance, operations, or your BI team has already agreed. Ask the agent to calculate it and retain the sources, filters, joins, date range, and timezone. If it cannot match the trusted result within an agreed tolerance and explain the difference, stop there.
2\. Make the semantic definitions visible
Terms such as active user, revenue, conversion, churn, and renewal often hide important choices. Confirm the definition, grain, denominator, exclusions, currency treatment, and source owner. A plausible chart built on the wrong definition is still a failed analysis.
3\. Reconcile across systems
Compare the answer with the source database, the governed BI dashboard, and the relevant finance or operational report. Investigate mismatches instead of averaging them away. Common causes include refresh timing, late events, duplicate rows, join logic, filters, refunds, and timezone boundaries.
4\. Apply least privilege
Start with read-only access to the smallest useful set of tables, columns, rows, and time periods. Keep analytical credentials separate from credentials that can write, send, purchase, or change records. Access controls reduce the blast radius, but they do not validate the analysis.
5\. Review the evidence, not just the answer
Inspect which records and documents were used, when they were updated, how calculations were performed, what was excluded, and where uncertainty remains. NIST's 2026 work on evaluation probes for agentic AI focuses on structured audit trails that connect agent decisions to supporting evidence. That is a useful standard even when the interface is simple.
NIST source: https://www.nist.gov/programs-projects/building-evaluation-probes-agentic-ai
6\. Keep approval between analysis and action
Let the agent prepare a recommendation, draft message, or proposed update first. Require a named human owner to review the evidence and approve any action that changes systems, contacts customers, moves money, or affects access. Log what was approved and what actually happened.
A practical decision rule:
Green: it reproduces known metrics, uses visible definitions, explains discrepancies, operates with narrow access, shows traceable evidence, and keeps consequential actions behind approval.
Yellow: it is useful for exploration, but its results still need manual reconciliation before a decision.
Red: it cannot show its work, silently changes definitions, asks for broader access than the task needs, or acts without a clear approval gate.
The product announcement is useful context, but its capability statements remain vendor claims.
OpenAI source: https://openai.com/index/put-data-to-work/
Which check would be most likely to expose a false sense of trust in your current data stack: metric reproduction, semantic definitions, reconciliation, evidence review, or action approval?