Beyond Keyword Matching

The simplest way to classify a support message is keyword matching. If the message contains "refund," route it to billing. If it contains "bug," route it to engineering. If it contains "cancel," route it to retention.

This works for about a week. Then you discover that "I tried to cancel my meeting but the button is bugged and I need a refund of my time" contains all three keywords and should be classified as a bug report, not a refund request.

Keyword matching breaks on real language because people do not write support messages like search queries. They write messy, contextual, multi-intent sentences.

How a Classification Model Thinks

A trained classification model reads the entire message and considers context, word relationships, and patterns it learned from thousands of similar messages. It does not look for keywords. It understands meaning.

"I was charged twice last month" maps to billing_dispute. Not because of the word "charged" but because the model recognizes the pattern of a customer describing an unwanted billing event.

"Your app crashes every time I open the settings page" maps to bug_report. Not because of the word "crashes" but because the model recognizes the structure of a reproducible issue description.

"Can I upgrade to the annual plan?" maps to subscription_upgrade. Not because of "upgrade" but because the model understands the customer is asking about changing their plan tier.

The 92% Number

Out of the box, with no customization, the model correctly classifies 92% of support messages into the right intent from a set of 315 possibilities.

What does 92% mean in practice? For every 100 messages:

• 92 are classified correctly and can trigger automated actions
• 8 are classified incorrectly, but the confidence score on those 8 is usually low enough that they get routed to humans instead of triggering wrong actions

This is why confidence thresholds matter. If you set your auto-response threshold to 85% confidence, the system only acts automatically on messages where it is highly confident. The ones it is unsure about go to your queue.

How to Use Classification in Your Workflow

Step 1: Every message gets classified. As soon as a customer sends a message, the classifier returns the intent and confidence score. This takes under 200 milliseconds.

Step 2: High-confidence messages trigger rules. If the intent is password_reset with 96% confidence, send the password reset link automatically. If the intent is bug_report with 91% confidence, create a GitHub issue and send an acknowledgment.

Step 3: Low-confidence messages go to humans. If the intent is billing_dispute with 62% confidence, the system is not sure enough to act. Route it to your support queue with the suggested classification for context.

Step 4: Review and improve. When you manually handle a message, the classification is either confirmed or corrected. Over time, you build a feedback loop that helps you understand where the model excels and where it struggles.

Tips for Getting the Most Out of Classification

Set your threshold based on risk. For a password reset (low risk if wrong), 80% confidence is fine. For a refund (higher risk if wrong), set it to 90%.

Start with auto-responses for your top 5 intents. Do not try to automate everything at once. Pick the 5 intents that come up most often and have the simplest responses. Expand from there.

Use the confidence distribution to find edge cases. If a lot of messages are landing in the 60 to 75% confidence range for a specific intent, look at those messages. You might discover that customers phrase things in ways the model finds ambiguous.

Trust the data over your intuition. The model processes hundreds of messages per day across thousands of businesses. Its patterns are broader than what you see in your own inbox. If the model consistently classifies something a certain way and customers are happy with the automated response, it is working.