The Air Canada Disaster

In 2022, Air Canada's chatbot told a grieving customer that he could book a full-fare flight and apply for a bereavement discount retroactively. That policy didn't exist. The chatbot made it up.

The customer booked the flight, applied for the discount, got denied, and sued. In February 2024, the Canadian tribunal ruled in the customer's favor. Air Canada argued that the chatbot was "a separate legal entity" responsible for its own words. The tribunal wasn't impressed.

Cost to Air Canada: CA$812.02 (CA$650.88 in damages, plus interest and tribunal fees) and the reputational hit of arguing that their own chatbot wasn't their responsibility. The case became the poster child for AI hallucination in customer support.

Why Chatbots Hallucinate

Large language models (the technology behind most AI chatbots) don't look up answers. They generate text that is statistically likely to follow the prompt. When a customer asks about a bereavement fare, the LLM generates text about bereavement fares based on patterns in its training data. If those patterns suggest bereavement discounts exist (because many airlines do offer them), the model generates a plausible-sounding policy even if your airline doesn't have one.

This is different from a database query. A database returns what's stored. An LLM returns what sounds right. "Sounds right" and "is right" overlap most of the time. When they don't, you have a hallucination.

RAG (retrieval-augmented generation) helps but doesn't eliminate the problem. RAG means the chatbot searches your docs before generating an answer. But if the docs are ambiguous, incomplete, or the chatbot can't find the right article, it falls back to generating text from its general training. That's where hallucinations creep in.

How to Prevent It

Use confidence thresholds. Most AI systems can report how confident they are in their response. Set a threshold (80-90%) below which the system says "I'm not sure about that. Let me connect you with a person" instead of guessing. This catches the cases where the AI doesn't have a clear match in your knowledge base.

Constrain the output space. Instead of letting the AI generate freeform text, give it a fixed set of possible responses. If the customer asks about refunds, the AI can only return your actual refund policy text, not generate a paraphrase of it. Paraphrasing is where details get distorted.

Add a fallback for unanswerable questions. If the query doesn't match any documented topic, the chatbot should say so and offer to connect the customer with a human. The Air Canada problem happened partly because the bot tried to answer a question it had no verified information about.

Use classification instead of generation. This is the most reliable approach. Instead of asking an AI to answer the question, ask it to identify the intent (refund request, shipping inquiry, password reset) and then trigger a pre-written response or action for that intent. The AI picks from known options rather than generating new text.

A purpose-built classifier like Supp takes this approach. It identifies the customer's intent from 315 possible categories. It doesn't generate text. It routes to a predefined action. Since it's picking from a fixed list rather than generating freeform answers, it can't hallucinate a policy that doesn't exist.

For Teams Using LLM-Based Chatbots

If you're using Intercom Fin, Zendesk AI, or another LLM-based tool, hallucination risk is inherent. You can reduce it:

Keep your knowledge base accurate and complete. The more coverage your docs have, the less the LLM needs to "fill in gaps" with generated content. Review articles quarterly.

Add explicit "the AI should NOT answer questions about" rules. Most platforms let you define topics the AI should refuse to handle. Add your high-risk topics: pricing (if it changes frequently), legal terms, compliance questions, and anything involving money.

Review AI-generated responses regularly. Sample 20-30 AI responses per week and verify accuracy. This catches drift before customers do.

Add disclaimers on sensitive topics. "This is general information. For specific policy questions, please contact our team directly." Not ideal, but it limits liability.

The Safe Middle Ground

The safest approach for most teams: use AI for classification and simple factual retrieval, not for generating policy interpretations or complex answers.

AI is great at: "This message is a refund request" (classification). "Your order #1234 shipped March 10 via USPS" (factual retrieval). "Our business hours are 9-5 Monday through Friday" (static FAQ). "I'll connect you with our billing team" (routing).

AI is risky at: "Based on our policy, you may be eligible for a partial refund if..." (interpretation). "I can offer you a 15% discount on your next order" (making commitments). "Your issue is caused by X and you should try Y" (technical diagnosis). "You can cancel anytime for a full refund" (stating terms that may not be accurate).

Keep AI in the "great at" column and you won't have an Air Canada moment.