Why Your Last Chatbot Failed, and It Was Not the Model
Most bad chatbots were measured on deflection rate, which pays them to avoid handing over. The metric caused the experience your customers hated.
The metric caused the experience
Most chatbot dashboards lead with deflection rate: the share of conversations that ended without a human.
Read that definition again from the customer's side. Every handover counts as a failure. Every refusal to hand over counts as a win. A customer who gave up and closed the window scores identically to a customer who was helped.
A system optimised against that number learns to keep people circling through restated questions and suggested articles. This is not a subtle misalignment. It is the precise mechanism behind the experience people are describing when they say they hate chatbots, and it was designed in on purpose, by whoever chose the metric.
Three other things that were probably true
| What went wrong | What it looked like to the customer | Fixable by |
|---|---|---|
| Scoped to answer everything | Confident wrong answers on edge cases | Defining what it may answer |
| No access to your systems | Paraphrasing the FAQ page | A read path into orders, stock, billing, calendar |
| Nobody read the transcripts | The same failure every week for a year | One person, one hour, weekly |
The second is the one that quietly determines everything. A support automation connected to nothing can only restate published content, so it competes with your own search box and loses, because the customer read that page before opening the chat.
The questions that generate contacts are specific and personal. Where is my order. Is this still available. Why was I charged this. Can I move my appointment. Answering any of them needs a read path into a real system, and building that path is most of the actual work.
A proposal that skips it is quoting for a wrapper around a knowledge base. The demo will look excellent, because demos ask general questions.
What a working version does differently
It answers only what it can verify, and it says where the answer came from.
In a brokerage deployment we ran, the automation answered what the listing itself could answer: floor, area, price, what is included, whether the property is still available. Everything else went to a named agent with the conversation attached. Response time went from 6 hours to 8 minutes, and the reason it worked is that the automation never guessed.
Anything binding goes to a person by design. Pricing outside the published rate, terms, commitments. That boundary is the subject of our rules for keeping a person in the loop, and it is not a limitation we apologise for: an automation that agrees something on your behalf at 2am is a liability rather than a feature.
Handover is the whole product
Three things go wrong here and all three are cheap to fix.
The escape hatch is hidden, so the customer has to guess a magic phrase to reach a person. Say in the first message that a human is available.
The handover arrives as a bare alert, so the agent opens by asking what the customer has already explained twice. Carry the transcript across, or the automation has cost time rather than saved it.
The queue behind the handover is not staffed for what the bot escalates, so a fast refusal becomes a long silence. That is a capacity decision, and it has to be made before launch rather than discovered in week two.
Route on the first sign of frustration, not the third. The cost of an unnecessary handover is a few minutes of an agent's time. The cost of a refused one is the customer.
When we say do not build it
More often in this category than in any other, and usually for one of three reasons.
The contacts are mostly things a bot cannot verify. The volume is too low for anyone to maintain it. Or the real problem is that human response is slow, and support automation would be a decoration over that.
The third case deserves naming because it is common. If enquiries wait four hours because nobody is there at seven in the evening, a bot that says something friendly and unhelpful at seven in the evening has not fixed the wait, it has automated it. The money is better spent on routing, on coverage, or on removing the reason people are contacting you at all.
That is the same test as the fifth readiness condition: if the bottleneck is not here, making this part faster changes nothing anyone can bank.
What to ask the next vendor
Which systems will it read from, and what will it do when that read fails.
What is it measured on, and if the answer is deflection rate, what happens to the number when it hands over correctly.
How does a customer reach a person, in how many messages, and who is waiting when they arrive.
And ask to see a transcript from a real deployment on a bad day. The order-processing split between rules and model is what a defensible answer to the first question looks like: deterministic questions answered by rules, unstructured input read by a model, and the two never swapped.
01What is actually wrong with measuring deflection rate?+
It pays the system to do the thing customers hate. Deflection rate counts conversations that ended without a human, so every handover is scored as a failure and every refusal to hand over is scored as a win, regardless of whether the customer got what they came for. A bot optimised against that number learns to keep people in the loop of restated questions and suggested articles, because a customer who gives up and closes the window counts identically to a customer who was helped. That is not a subtle misalignment; it is the exact mechanism behind the experience most people are describing when they say they hate chatbots. Measure resolution instead, and measure time to handover, and the same technology produces an entirely different experience because the incentive now points the same way the customer does.
02Our bot could only repeat the FAQ page. Why?+
Because it had no access to the systems that hold the answers, which is a scoping and integration decision rather than a model limitation. A support automation connected to nothing can only paraphrase published content, so it is competing with your own search box and losing, because the customer has usually already read that page before opening a chat. The questions that actually generate contacts are specific and personal: where is my order, is this still available, why was I charged this, can I move my appointment. Answering those requires a read path into the order, inventory, billing or calendar system, and building that path is most of the real work. Any proposal that skips it is quoting for a wrapper around a knowledge base, and the demo will look excellent because demos ask general questions.
03How should the handover actually work?+
Immediately, visibly, and with the whole conversation attached rather than as a fresh ticket. Three things go wrong in practice and all three are fixable. The escape hatch is hidden, so the customer has to guess a magic phrase to reach a person, which converts mild irritation into anger. The handover arrives as a bare alert, so the agent opens with a question the customer has already answered twice and the automation has just cost time rather than saved it. And the queue behind the handover is not staffed for the volume the bot escalates, which turns a fast refusal into a long silence. Say plainly in the first message that a person is available, route on the first sign of frustration rather than the third, and carry the transcript across.
04When is the honest answer that you should not deploy one at all?+
When the contacts you receive are mostly things a bot cannot verify, when the volume is too low for anyone to maintain the thing, or when the real problem is that your human response is slow and support automation would be a decoration over that. The last case is common and worth naming: if enquiries wait four hours because there is nobody to answer them at seven in the evening, a bot that says something friendly and unhelpful at seven in the evening has not fixed anything, it has just made the wait feel automated. In that situation the money is better spent on routing, on coverage, or on removing the reason people are contacting you at all. We say no to support automation on these grounds more often than to any other category of project.
