What to do when AI agents hallucinate? The answer is more about architecture than prompting.

In the last few weeks, I deployed an AI agent into production with a simple goal: to warm up leads and provide support for a medical clinic, scheduling appointments and answering questions directly on WhatsApp.

In testing, everything went smoothly.. but reality hit hard when actual leads started coming in.

I began tracking the success rate of tasks and monitoring conversations directly in the database and that’s when inconsistencies emerged.

Even with a well-structured knowledge base, the agent started making things up.

My first reaction was to improve the knowledge base. It helped, but it didn’t solve 100% of the issues.

Then, I refined the prompt. It improved again, but in some cases, it still went off the rails.

The turning point was breaking down the solution: a single agent just wasn’t cutting it.

The solution was to place a “classifier” agent at the front, which identifies the user’s intent and routes them to other specialized agents.

Theory vs. Practice

This practical experience made me very curious, so over the weekend, I stopped to understand the theory behind hallucinations.

I ended up stumbling upon an article (I can’t remember who recommended it, but it was enlightening).

I already knew that hallucination isn’t a “bug,” but rather an inherent and inevitable feature of LLMs.

The focus shouldn’t be on eliminating it, but rather on mitigating and managing it. However, in practice, this is quite complicated.

Based on my experience and the lessons from that article, here are 3 practical tips I learned during this process:

1. Treat your knowledge base like a product

The main cause of hallucinations comes from bad, incomplete or outdated data.

Improving the knowledge base (a technique known as RAG) is the first step to grounding the AI in facts, reducing the chance of it inventing information.

2. Be obsessed with the prompt

Vague instructions generate vague answers.

Studies show that how you ask directly impacts the quality of the answer. Define the agent’s role, give clear examples and restrict the scope of the task. A good prompt is a good start to avoid deviations.

3. For complex problems, use multiple agents

A single agent trying to handle many different tasks (scheduling, answering questions, qualifying) can get confused. The solution that worked for me was using a router agent to identify the intent and pass the ball to a specialist agent.

This approach creates natural “guardrails” in the system, aligning with what the article calls control mechanisms.

I talk about many of these lessons almost weekly to various people, but when you are the one implementing it.. you end up forgetting, learning the hard way and then truly learning :)

The study is dense, but it’s worth a look for anyone building with AI who wants to go beyond the surface level.

This article was originally posted in Portuguese on my LinkedIn. You can read the original version and follow me there for more insights.