The AI agent mirage

The demo lasted twelve minutes and was flawless. The agent read an inbox of invoices, matched them against bank payments and drafted a plain-language summary for leadership. The board applauded and approved a pilot. Three months later, the pilot is still on a test server. It works with the sample data. With real data, it gets things wrong in ways nobody can explain, and nobody dares let it write to the ERP.

The industry has a name for that place: “pilot purgatory”, where AI projects shine in a presentation but never reach operations. Sometimes the model fails on real cases. Sometimes data, tools, permissions or a clear process definition are missing. Measure each part before blaming just one.

The error, now at scale

Manual reconciliation can fail without anyone noticing. If an agent applies an ambiguous rule to thousands of transactions without controls, it too can repeat an error at scale before anyone catches it. The cause may lie in the process definition, data, tools or model. Test each part and review exceptions before giving it volume.

AI doesn't organize a confused process on its own. Ungoverned spreadsheets, knowledge held by one person and five versions of the truth raise the risk of wrong answers delivered in convincing prose.

The model is the engine. Without a road, a bigger engine just makes more noise.

What the demo doesn't show

Moving from a demo to reliable operations means testing the model and the system around it. These five pieces often determine whether a pilot can progress.

• Written rules: what to do, what not to do and what to do when unsure. Without them, the agent improvises.
• Connected, defined data: access to the real sources, not last week's export, with the definitions the business uses.
• Role-based permissions: which tools it uses, what it reads and what it may write, just like a person. The collections agent doesn't need to see payroll.
• A complete record: what it proposed, with what data, which rule it applied. Without traceability, nobody trusts it and nobody can audit it.
• Human approval where it matters: high-impact actions wait for a sign-off. That doesn't slow the agent down; it's what lets you give it real work.

The right order

A prudent sequence starts by writing the process rules and exceptions, connecting trustworthy sources and using tested rules for work that doesn't need a model. The remaining cases may call for conventional software, an agent or human review. Choose among them with tests, clear limits and approval proportionate to risk.

It's less spectacular than twelve minutes in front of the board. It also shows which part of the process can operate safely and which part needs more work.

The scenes in these episodes are composites of common situations, not customer cases. Any figures in them are illustrative.