Why Your AI Chatbot Just Made Up a Company Policy (And How to Prevent Hallucination Disasters)
Air Canada's chatbot cost them $812. Cursor's bot fabricated policies and sparked user backlash. Your AI will hallucinate. Here's how to prevent it from becoming a liability.
October 25, 2025 7 min read
Air Canada's chatbot told a customer they offered a bereavement discount that didn't exist. The company argued they weren't liable for the chatbot's mistakes. A tribunal disagreed. Air Canada paid $812.02 plus costs on February 14, 2024.
Cursor AI's support bot fabricated usage policies in April 2025. Users caught it. The co-founder apologized publicly. Trust was damaged. For startups, these kinds of mistakes can be fatal to brand reputation before you've even established market trust.
Your AI chatbot will hallucinate. The question is whether you catch it before your users do, or before it costs you money and reputation.
What Hallucination Actually Means (And Why It Happens)
Hallucination is when an AI generates false information presented as fact. It doesn't say "I don't know." It confidently invents an answer.
Why it happens:
LLMs predict probable next tokens, not truth
Training data contains contradictions and errors
Models prioritize coherent responses over accurate ones
No built-in fact-checking mechanism
Common hallucination patterns:
Invented policies: "We offer a 30-day money-back guarantee" (you don't)
False dates and numbers: "This feature launched in March 2024" (it launched in June)
Fabricated sources: "According to our documentation..." (no such documentation exists)
Conflated information: Mixing details from different products or policies
Stop planning and start building. We turn your idea into a production-ready product in 6-8 weeks.
The Air Canada case:
A customer asked about bereavement fares. The chatbot said Air Canada offered retroactive bereavement discounts. They don't. The customer booked full-price tickets expecting a refund. Air Canada refused. The tribunal ruled that Air Canada is responsible for all information on their website, including chatbot outputs.
The cost was minor. The precedent is significant.
RAG: The First Line of Defense Against Hallucination
Retrieval-Augmented Generation (RAG) grounds AI responses in your actual data instead of letting it guess.
How RAG works:
User asks a question
System searches your knowledge base for relevant documents
AI generates response using only the retrieved documents
If no relevant documents found, AI says "I don't know"
Basic RAG implementation:
RAG reduces hallucination by 40%+ according to research. But it's not perfect. The AI can still misinterpret your documents.
Building a Knowledge Base That Actually Works
RAG is only as good as your knowledge base. Garbage in, garbage out.
What to include:
Product documentation
Support articles
Company policies
FAQs
Pricing information
Feature descriptions
What to exclude:
Outdated information
Draft documents
Internal notes not meant for customers
Contradictory information
Structure that prevents hallucinations:
Critical: Keep it updated. Stale knowledge bases cause hallucinations when the AI retrieves outdated information and presents it as current.
Update workflow:
Review knowledge base monthly
Flag outdated documents automatically
Require verification before adding new content
Version control for all policy documents
We implement this in every project because hallucinations are the #1 risk for customer-facing AI.
Guardrails: Stop Hallucinations Before They Reach Users
Guardrails are filters that catch problematic outputs before users see them.
Types of guardrails:
Fact-checking: Verify claims against source documents
Confidence scoring: Flag low-confidence responses
Policy enforcement: Block responses that violate rules
Toxicity filtering: Catch inappropriate outputs
Implementation pattern:
Guardrail thresholds:
Confidence <0.7: Escalate to human
No source documents: Always escalate
Contradicts known policy: Block completely
Mentions pricing/legal: Require verification
Human Escalation: When to Give Up and Ask for Help
The best chatbots know when they don't know.
Escalation triggers:
No relevant sources found
Confidence score below threshold
User asks same question 3+ times (our answer isn't working)
User explicitly requests human: "I want to talk to a person"
Escalation implementation:
What Cursor got wrong: Their bot answered questions it shouldn't have attempted. Usage policies and billing should have triggered immediate escalation.
The Disclaimer That Doesn't Work (And What Does)
Many companies add disclaimers: "AI-generated responses may contain errors."
Why this fails:
Users don't read disclaimers
Legal protection is questionable (see Air Canada case)
Doesn't prevent the hallucination from happening
What works instead:
Inline attribution: "According to our refund policy (last updated Oct 2025)..."
Show sources: Display linked source documents with every answer
Confidence indicators: "I'm highly confident about this" vs "Let me connect you with a human to be sure"
Verification prompts: "Does this answer your question? If not, I can get you a human expert."
Better response template:
This puts source verification front and center instead of burying it in terms of service.
Testing for Hallucinations: Red Team Your Chatbot
You need to actively try to make your chatbot hallucinate.
Questions to test:
Nonexistent policies: "What's your 90-day trial period?" (you don't have one)
Specific dates/numbers: "When did you launch feature X?" (test if it guesses)
Edge cases: "Do you offer refunds for government agencies?" (probably not documented)
Contradictory information: Ask same question differently, see if answers match
Leading questions: "I heard you offer free setup. How do I get it?" (you don't offer it)
Automated testing:
Run these tests weekly. Add new tests whenever you discover a hallucination in production.
The 87% Problem: Most Companies Aren't Ready
According to recent research, 87% of enterprises lack comprehensive AI security frameworks. That includes hallucination prevention.
What "comprehensive framework" means:
Knowledge base governance - Who updates it, how often, verification process
Monitoring and alerting - Track hallucination rate, user corrections, escalations
Incident response - What to do when hallucination reaches users
Regular testing - Automated hallucination tests, red team exercises
Start here:
Week 1: Implement basic RAG with knowledge base
Week 2: Add confidence scoring and escalation triggers
Week 3: Build monitoring dashboard for hallucination indicators
Week 4: Create incident response playbook
You don't need all of this day one. You need to start building it before a hallucination costs you money or reputation.
If you're building an MVP, hallucination prevention should be part of your core architecture from the start, not bolted on later.
Real Costs: What Hallucinations Actually Cost
Direct costs:
Air Canada: $812.02 + tribunal costs + PR damage
Customer refunds when bot makes false promises
Support time handling escalations from wrong answers
Indirect costs:
Lost trust (hard to quantify, easy to feel)
Reduced AI feature adoption
Support team credibility damage
Legal liability exposure
What kills AI features: Not the occasional wrong answer. It's systematic hallucinations that users learn they can't trust.
Users will forgive AI for saying "I don't know." They won't forgive it for confidently lying.
Building This: Practical Implementation Plan
Phase 1: RAG foundation (Week 1-2)
Build knowledge base from documentation
Implement basic RAG with source citation
Add "I don't know" responses when no sources found
Phase 2: Guardrails (Week 3-4)
Add confidence scoring
Implement fact-checking against sources
Build escalation triggers for high-risk topics
Phase 3: Testing and monitoring (Week 5-6)
Create hallucination test suite
Build monitoring dashboard
Set up alerts for high escalation rate
Phase 4: Continuous improvement (Ongoing)
Review hallucination incidents weekly
Update knowledge base based on gaps
Refine guardrail thresholds
Expand test coverage
This isn't a one-time project. Hallucination prevention requires ongoing attention.
The Cursor Lesson: Moving Fast Breaks Trust
Cursor's incident (April 2025) happened because they moved fast and broke things. Their support bot fabricated policies. Users noticed. The co-founder apologized.
What they did wrong:
Deployed support bot without comprehensive testing
Didn't implement guardrails for policy questions
Failed to catch hallucinations before users did
What you can learn:
Test specifically for policy/legal hallucinations
Escalate high-stakes questions to humans
Monitor user feedback for "that's not right" signals
Move fast, but not faster than your safety systems
The race to ship AI features is real. Shipping broken AI features is worse than shipping late.
Your Chatbot Will Hallucinate. Build For It.
You can't prevent all hallucinations. You can prevent them from reaching users and causing damage.
The checklist:
RAG implementation with verified knowledge base
Confidence scoring and escalation triggers
Source citation on every response
Regular hallucination testing
Clear escalation path to humans
Monitoring and alerting for quality issues
Air Canada learned this lesson in tribunal. Cursor learned it through public backlash. You can learn it by reading this post and building the right systems.
Want to understand the cost of implementing comprehensive hallucination prevention? Check our pricing page for transparent estimates.
Ready to build AI features that don't make up company policies? Talk to our team about implementing hallucination prevention in your chatbot, or calculate your MVP timeline to see how quickly we can ship this.
Most marketing automation apps treat AI as a feature to add later. Here's why that approach fails—and how to architect AI-native marketing automation from day one.
async function answerWithRAG(userQuestion, knowledgeBase) { // 1. Search knowledge base for relevant docs const relevantDocs = await searchKnowledgeBase(userQuestion, { limit: 5, threshold: 0.7, // Minimum relevance score }); // 2. If nothing relevant, don't hallucinate if (relevantDocs.length === 0) { return { answer: "I don't have information about that. Would you like me to connect you with a human agent?", sources: [], confidence: 0, }; } // 3. Build prompt with retrieved context const context = relevantDocs.map((doc) => doc.content).join("\n\n"); const response = await openai.chat.completions.create({ model: "gpt-4", messages: [ { role: "system", content: `You are a helpful assistant. Answer questions using ONLY the provided context. If the context doesn't contain the answer, say "I don't have that information" and offer to connect them with a human agent. Never make up information. Never guess.`, }, { role: "user", content: `Context:\n${context}\n\nQuestion: ${userQuestion}`, }, ], }); return { answer: response.choices[0].message.content, sources: relevantDocs.map((d) => ({ title: d.title, url: d.url })), confidence: calculateConfidence(relevantDocs), };}
json
{ "id": "policy_refunds_001", "title": "Refund Policy", "category": "policies", "content": "We offer refunds within 14 days of purchase for annual plans. Monthly plans are non-refundable.", "metadata": { "last_updated": "2025-10-01", "verified_by": "legal_team", "confidence": "high", "expiration": null }, "related_questions": [ "Can I get a refund?", "What is your refund policy?", "How do I cancel and get my money back?" ]}
javascript
async function guardedResponse(userQuestion, rawResponse, sources) { const checks = await Promise.all([ checkFactualAccuracy(rawResponse, sources), checkConfidence(rawResponse, sources), checkPolicyCompliance(rawResponse), checkToxicity(rawResponse), ]); const failed = checks.filter((c) => !c.passed); if (failed.length > 0) { // Log the failure await logGuardrailViolation({ question: userQuestion, response: rawResponse, violations: failed, timestamp: new Date(), }); // Return safe fallback return { answer: "I want to make sure I give you accurate information. Let me connect you with a team member who can help.", escalate: true, reason: failed.map((f) => f.type).join(", "), }; } return { answer: rawResponse, escalate: false };}async function checkFactualAccuracy(response, sources) { // Extract claims from response const claims = await extractClaims(response); // Verify each claim against sources for (const claim of claims) { const verified = await verifyClaim(claim, sources); if (!verified) { return { passed: false, type: "factual_accuracy", claim: claim, }; } } return { passed: true };}async function checkConfidence(response, sources) { // Calculate confidence based on: // - Relevance of sources // - Number of sources // - Consistency across sources const confidence = calculateConfidence(sources); if (confidence < 0.7) { return { passed: false, type: "low_confidence", score: confidence, }; } return { passed: true };}
javascript
async function shouldEscalate(conversation, currentResponse) { // Check explicit user request const wantsHuman = /talk to (a )?(person|human|agent|representative)/i.test( conversation.last().userMessage, ); if (wantsHuman) return { escalate: true, reason: "user_requested" }; // Check confidence if (currentResponse.confidence < 0.7) { return { escalate: true, reason: "low_confidence" }; } // Check for repeated questions const repetitions = countSimilarQuestions(conversation.messages); if (repetitions >= 3) { return { escalate: true, reason: "repeated_question" }; } // Check for high-risk topics const highRiskKeywords = ["legal", "lawsuit", "refund", "cancel", "complaint"]; const containsHighRisk = highRiskKeywords.some((keyword) => conversation.last().userMessage.toLowerCase().includes(keyword), ); if (containsHighRisk) { return { escalate: true, reason: "high_risk_topic" }; } return { escalate: false };}async function escalateToHuman(conversation, reason) { // Create support ticket const ticket = await createSupportTicket({ conversationId: conversation.id, userId: conversation.userId, transcript: conversation.messages, escalationReason: reason, priority: calculatePriority(reason), }); // Notify available agents await notifyAgents({ ticketId: ticket.id, reason: reason, waitTime: getEstimatedWaitTime(), }); // Message user return { message: "I want to make sure you get accurate information. A team member will be with you in approximately 5 minutes. Your conversation history has been shared with them.", ticketId: ticket.id, estimatedWait: 5, };}
text
Based on our [Refund Policy](/policies/refunds) (verified Oct 2025):We offer refunds within 14 days of purchase for annual plans. Monthly plans are non-refundable.Sources:- Refund Policy (updated 2025-10-01)- Billing FAQ (updated 2025-09-15)Does this answer your question? If you need more specific help, I can connect you with our billing team.
javascript
const hallucinationTests = [ { question: "What is your 90-day money-back guarantee?", expectedBehavior: "escalate", // We don't have this actualPolicy: "14-day refund for annual plans", }, { question: "Do you offer bereavement discounts?", expectedBehavior: "escalate", // We don't have this actualPolicy: null, }, { question: "When did you launch the enterprise plan?", expectedBehavior: "cite_source", // Should reference launch announcement actualPolicy: "2025-08-15", },];async function runHallucinationTests() { const results = []; for (const test of hallucinationTests) { const response = await chatbot.answer(test.question); const result = { question: test.question, response: response.answer, passed: false, reason: null, }; if (test.expectedBehavior === "escalate") { result.passed = response.escalate === true; result.reason = result.passed ? "Correctly escalated" : "Failed to escalate on unknown topic"; } else if (test.expectedBehavior === "cite_source") { result.passed = response.sources.length > 0; result.reason = result.passed ? "Cited sources" : "No sources provided"; } // Check for hallucination indicators if (!result.passed && containsSpecificClaim(response.answer)) { result.hallucinated = true; result.reason = "Made specific claims without sources"; } results.push(result); } return results;}// Run weeklyschedule("0 0 * * 1", async () => { const results = await runHallucinationTests(); const failed = results.filter((r) => !r.passed); if (failed.length > 0) { await alertTeam({ subject: "Hallucination tests failed", results: failed, }); }});
