AI agents in production: what nobody tells you before implementing them

AI agents are the topic of the moment. Every week a new platform appears promising to deploy a team of autonomous agents that will transform your business in days. The reality is more nuanced, more interesting and more complex than any demo you have seen.

This article does not try to sell you AI agents. It tries to give you enough technical judgement to make an informed decision about whether it makes sense to implement them in your organisation right now, in which processes and with what realistic expectations.

What an AI agent really is and how it differs from conventional automation

Conventional automation follows a deterministic flow: if A occurs, execute B. If C occurs, execute D. It is predictable, reproducible and auditable. Its limitation is that it cannot handle situations not anticipated in the design.

An AI agent adds a reasoning layer: it receives an objective, observes the context, decides which tools to use and in what order, executes actions and adjusts its behaviour based on results. It can handle variability that conventional automation cannot.

When an AI agent makes sense (and when it doesn't)

It makes sense when the process requires:

• Interpreting unstructured natural language (emails, reviews, tickets, contracts)
• Making decisions with criteria that vary according to context
• Dynamically coordinating multiple tools or systems
• Handling exceptions that would be difficult to code as explicit rules

It does not make sense when the process is:

• Completely deterministic and without contextual variability
• High-criticality where errors have serious consequences
• Dependent on scarce or very low-quality data
• A new process without sufficient history to evaluate agent performance

The real costs nobody mentions in demos

AI agent demos are impressive. The real operating costs of maintaining them in production are too, but in the opposite sense. The main cost factors underestimated in the proposal phase are LLM API token costs — which can be 5 to 50 times more expensive than a conventional API call for the same volume of operations — plus response latency, the engineering cost of designing system prompts and fallback flows, and the cost of continuous monitoring.

The evaluation problem in production

With conventional automation, knowing if it is working correctly is straightforward. With an AI agent, evaluation is fundamentally more complex. Designing a robust evaluation system for agents in production requires:

• An evaluation dataset with representative cases and expected responses
• Domain-specific metrics (not just generic "accuracy")
• Random sampling of decisions for periodic human review
• Automatic alerts to detect performance degradation

The three deployment models and which to choose

Fully autonomous agent: acts without human intervention. Maximum operational efficiency, maximum risk. Only recommended for low-risk processes with robust continuous evaluation.

Agent with human-in-the-loop: the agent proposes actions that a human approves before executing. Significantly reduces risk at the cost of efficiency. The most recommended model for processes that directly impact customers or finances.

Agent as operator assistant: the agent does not execute actions but provides context, recommendations and drafts that the human decides whether to use. Lowest risk and easiest to adopt. An excellent way to start.