Why script based chatbots fall behind
Older chatbots run on decision trees: if the user presses one, show this answer. That is reliable for very narrow flows but has two fatal flaws. First, every change in price, policy, or product requires manual script editing. Second, real users rarely phrase questions the way you anticipated.
The result is a bot that looks smart in a demo and frustrates in production, and eventually every question gets thrown to a human anyway.
What RAG is and why it differs
RAG stands for retrieval augmented generation. The idea is simple: instead of the language model answering from its own memory, the system first retrieves relevant document fragments from your knowledge base, then asks the model to compose an answer using only those fragments.
- Answers come from your documents, not from general internet knowledge.
- Updating knowledge means updating documents, not retraining a model.
- Answers can be traced back to their source passage.
- Hallucination risk drops, though not to zero, because the model is constrained to the supplied context.
Document quality determines answer quality
- One topic per section. Long documents mixing returns, shipping, and warranty in one paragraph cannot be split cleanly.
- Write in question and answer form for frequently asked information. Similarity between user phrasing and document content improves retrieval accuracy.
- Use customer vocabulary, not only internal terms. If customers say shipping cost and your document says logistics fee, retrieval can miss.
- Avoid complex tables and screenshots. Important information should exist as text.
- Date your policies. This prevents the bot answering with an outdated policy that was never removed.
Setting boundaries: what the bot may answer
- Safe for the bot: product information, opening hours, general policies, technical steps, and order status when connected to a system.
- Must go to a human: angry complaints, refund requests, price negotiation, legal or medical questions, and anything touching safety.
- Needs an honest I do not know: every question with no basis in the documents. A bot that defers to an agent beats a bot that invents.
Measuring whether the bot is useful
- Self service resolution rate, the share of conversations closed without escalation.
- Appropriate escalation rate. A bot that never escalates is suspicious.
- Answer accuracy on periodic samples, judged by humans against source documents.
- Customer wait time before and after deployment, the most immediately felt benefit.
- Unanswered questions, which should become your documentation backlog.
Most common implementation mistakes
- Uploading every document at once. Start with the ten to twenty most frequent questions, which cover most of the volume.
- Providing no escape to a human. This is the main cause of customer anger at bots.
- Letting documents go stale. A knowledge base with no owner is outdated within months.
- Measuring success by conversations handled rather than by satisfactory resolutions.
Frequently asked questions
Is our business data used to train the model?
In a RAG approach your documents are not used to retrain the model, only retrieved as context at answer time. Even so, review the data storage and processing policy of whichever provider you use.
How many documents are needed to be useful?
Far fewer than most people assume. Documents answering the twenty most frequent questions usually cover the bulk of repetitive conversations.
Can the bot answer in several languages?
Yes, though quality is best when documents exist in the same language your customers use.
How do you stop the bot making wrong promises?
Limit scope, write policies explicitly, and forbid the bot from stating figures absent from documents, such as discounts or specific delivery dates.
Next step
Collect the twenty most frequent questions from the last three months of conversations, write clean answers as documents, then run the bot only for that scope. Expand once accuracy is stable. Learn the implementation at WhatsCRM Hub.
