An AI Agent That Answers Customer Questions Correctly

Start with an inventory of what customers really ask

Before choosing a model or writing a single instruction, we gather the questions. Not the ones on your FAQ page, which someone wrote by guessing, but the ones sitting in your inbox, your call notes, your reviews and your staff’s memory.

We ask for a few weeks of real messages with personal details removed and sort every question into a category. The category decides how the agent handles it, and that matters far more than how polished the wording is.

The inventory also sizes the project honestly. If most questions are facts and policy, a grounded answer engine covers them. If status questions dominate, the job is really an integration project with a conversational front end, and it should be scoped that way.

How the agent builds an answer it can back up

A language model left to itself will answer anything, fluently and sometimes wrongly. The architecture we use, commonly called retrieval-augmented generation, narrows it down:

1. Your documents are split into short, self-contained passages and indexed for search, each tagged with its source and the date it was last reviewed.
2. When a customer asks something, the agent searches that index for the passages most likely to hold the answer.
3. The model is instructed to answer only from those passages and to name or link the page it drew on.
4. If the best passages don’t actually answer the question, it says so instead of filling the gap from general knowledge.

Two design choices make a large difference in practice. The first is precedence. When two documents disagree, say an old blog post and the current policy page, the agent needs a rule for which one wins. We flag certain documents as authoritative and remove stale ones rather than hoping the model notices a date. The second is passage quality. A policy split mid-sentence yields confident half-answers, so we split on headings and meaning, and we rewrite a handful of key documents into plain question-and-answer form when the originals are too tangled to retrieve well.

Linking the source has a practical payoff beyond trust. A customer who can click through to the exact policy page is less likely to argue with the answer, and your staff can check in seconds where any reply came from.

Teaching it to say “I don’t know” without losing the customer

The most valuable behavior in a question-answering agent is declining gracefully. A wrong answer about a return window or a warranty costs more than no answer, because the customer acts on it.

Compare two replies to “Do you service tankless water heaters from brand X?” when your documents never mention that brand:

The second reply turns a knowledge gap into a qualified inquiry. We build that pattern in on purpose. When retrieval confidence is low, the agent asks one clarifying question. If it still cannot ground an answer, it offers a handoff and, with the customer’s permission, records their contact details and the exact question. That question then shows up on the weekly gap report, so the answer can be added to your knowledge base.

We also test for the opposite failure: an agent so cautious that it refuses questions it could have answered. The evaluation set deliberately includes easy questions for exactly that reason, and an agent that ducks them fails the test just as surely as one that invents things.

Questions that need live data: orders, appointments and accounts

“Where is my order?” and “What time is my appointment?” can’t be answered from documents. They require a lookup in a system of record, which raises a security question: how does the agent know it is talking to the right person?

• Verification before disclosure. The agent asks for two pieces of information that must match, such as an order number plus billing ZIP, or a confirmation code plus the email on file. It never hints whether a partial match exists.
• Read-only by default. Lookup tools can read status. Changing or cancelling goes through a separate tool with its own confirmation step, or stays with staff entirely.
• Minimum disclosure. The tool returns only the fields the answer needs, such as status, date and window, not the full customer record, so the model never holds data it could leak.
• Attempt limits. Repeated failed verifications are throttled and logged, which stops anyone from guessing their way through order numbers.

These lookups are built through our AI integration work and connect to the systems you already run, whether that is an online store, a booking calendar or dispatch software.

When a question belongs to a person, and how it gets there

Some questions should never be finished by software. We write explicit escalation rules with you and give each one a test case:

• Complaints, damage claims, refunds and billing disputes.
• Anything involving safety, such as a gas smell, an electrical hazard or a medical symptom, where the agent gives your published emergency instruction and nothing more.
• Legal threats and requests about a customer’s personal data.
• Signs of frustration: repeated rephrasing, all caps, “this isn’t helping.”
• A direct request for a human, honored immediately rather than after three more questions.

The handoff itself shouldn’t feel like a dead end. During staffed hours the conversation can transfer to your live chat or trigger a callback; outside them, the agent says plainly when a person will follow up. Either way, your staff receive a summary of the question, what the agent already said and any details the customer supplied, so nobody has to ask the customer to repeat themselves.

Unanswered questions from prospects, as opposed to existing customers, are treated as leads rather than tickets. The agent asks whether the person would like a call back, and the question travels with their contact details so your sales team starts the conversation already knowing what was asked.

Keeping answers accurate after launch

Accuracy decays as the business changes. A policy gets revised, a holiday schedule is posted, a product is discontinued. We put four habits in place so the agent keeps pace.

A named knowledge owner

One person on your team is responsible for the source-of-truth documents. When they update a policy, the index refreshes automatically.

Expiring notices

Short-lived messages such as holiday closures or weather delays go into a notice file that the agent checks first and that carries an end date, so the notice never outlives the event.

The gap report

Each week you receive the questions that went unanswered, clustered by theme with counts. Most weeks the fix is a sentence or two added to a document.

Regression testing

Every real question the agent once got wrong joins a saved test set. After any change to knowledge, instructions or model, the full set is re-run and compared, so improving one answer never quietly breaks another.

Alongside the gap report, a short monthly summary shows question volume by category, the share resolved without a person and how quickly handoffs were picked up. Those three numbers are usually enough to decide where the next round of improvement should go.

Why a generic FAQ bot usually plateaus

Plugins that answer from an uploaded FAQ handle the first category in the inventory well. They tend to stall on the rest. They can’t verify a customer before a lookup, they have no notion of an authoritative document, they rarely produce a usable gap report, and testing them means chatting with them by hand. For a small site with a short list of stable questions, that is perfectly fine.

Once questions involve policies with consequences, live data or escalation rules, the difference comes from engineering those pieces properly, which is the work behind our AI agent and AI development services. If you are weighing where an agent should sit across your whole site, not just how it answers, the companion guide on how an AI agent fits a business website covers placement, speed and lead capture.

Related services

Frequently asked questions