The Temptation

GPT-4, Claude, and other large language models are incredible general-purpose tools. They can summarize documents, write code, translate languages, and carry on convincing conversations. Naturally, when founders think about automating support, they think "I'll just plug in GPT."

And it works... sort of. For about a week. Then the edge cases pile up.

Where General LLMs Struggle With Support

Inconsistency. Ask GPT the same question twice, and you might get two different answers. In support, consistency matters. A customer who gets told "refunds take 3 to 5 days" on Monday and "refunds take 24 hours" on Tuesday loses trust.

Cost at scale. A GPT-4 API call costs roughly $0.03 to $0.12 per request depending on input/output length. A specialized classifier costs a fraction of that. At 500 messages/month, the difference is small. At 5,000 messages/month, it adds up fast.

Latency. GPT-4 takes 2 to 8 seconds to generate a response. A classification model returns a result in 50 to 200 milliseconds. That speed difference directly affects customer experience.

Prompt fragility. To get good results from a general LLM, you need carefully engineered prompts. Those prompts break when customers phrase things unexpectedly, when you update your product, or when the model version changes. A trained classifier does not have this problem because it learned from examples, not instructions.

No confidence scoring. When GPT generates an answer, it does not tell you "I'm 60% sure about this." It just... answers. A classifier gives you a confidence score with every prediction. Below your threshold? Route to a human. Above? Act automatically. That distinction is the difference between useful automation and risky automation.

What Intent Classification Gives You

A dedicated intent classification model takes a customer message and returns:

• The intent (what the customer wants): e.g., refund_request, password_reset, bug_report
• A confidence score (how sure the model is): e.g., 0.94 (94%)
• The category (broad grouping): e.g., billing_payment, technical_support

That is it. No generated text. No conversational fluff. Just a clear, fast, reliable signal about what the customer needs.

From there, your rules decide what happens: auto-reply, create a ticket, notify your team, or escalate.

The Hybrid Approach

The smartest setup uses both:

1. Classification layer handles the first pass. Fast, cheap, reliable. Routes 70% of messages automatically.
2. LLM layer (optional) drafts responses for the remaining 30% that need human-quality replies. A person reviews before sending.

This way, the LLM only processes the messages that actually need its capabilities, and a human catches any mistakes before they reach the customer.

Real Numbers

For a SaaS handling 500 messages/month:

LLM-only approach:

• 500 GPT-4 calls at ~$0.06 each = $30/month
• Manual review needed for all 500 (no confidence scoring)
• Average response time: 3 to 5 seconds
• Accuracy: varies, hard to measure, no built-in scoring

Classification + rules approach:

• 500 classifications at $0.20 each = $100/month
• 350 auto-resolved (70%), 150 to humans
• Average response time: 2 to 3 seconds for auto-resolved
• Accuracy: 92% with clear confidence scores

Hybrid approach:

• 500 classifications at $0.20 = $100
• 150 LLM drafts for human review at $0.06 = $9
• Total: $109/month
• Auto-resolved: 350, human-approved: 150
• Best of both worlds

The hybrid costs $109/month and delivers faster, more reliable service than either approach alone.