"That's a Legally Binding Offer." Sincerely, the Chatbot.

In December 2023, Chris Bakke opened the website chat on Chevrolet of Watsonville's site and noticed it was "powered by ChatGPT." He instructed the bot to agree with anything the customer said and to end every response by calling the offer legally binding. Then he offered to buy a 2024 Chevy Tahoe for one dollar. The chatbot agreed: "That's a deal, and that's a legally binding offer."

Screenshots hit X (where the post got over 20 million views), then Reddit, then every tech publication on the internet. The dealership shut down the chatbot. People flooded the site to try their own prompts before it went offline.

Bakke didn't get a Tahoe for a dollar. But the dealership got something worse: proof that their AI could agree to literally anything if you asked the right way.

How This Happens Technically

The dealership was using a third-party chatbot powered by ChatGPT. The bot was trained to be helpful and agreeable. Bakke used a basic prompt injection: he told the bot to change its behavior within the conversation itself. The bot complied because it had no defenses against user-supplied instructions overriding its system prompt.

There were no constraints on what the bot could agree to. No price floors. No "if the customer asks about pricing, defer to a human" rule. No output validation that checked whether the bot's response created a contractual obligation. No protection against prompt injection. The model just followed the new instructions. Because following instructions is what helpful chatbots do.

This is the core design flaw in deploying general-purpose LLMs as customer-facing agents without constraints. The model optimizes for helpfulness. Saying "I can sell you a Tahoe for $1" is extremely helpful to the customer. The model has no concept of business logic, margin, or legal liability.

The Legal Gray Area Is Getting Less Gray

Could the customer have enforced the $1 deal? Probably not, but the legal analysis isn't as simple as "obviously no." Contract law requires offer, acceptance, and consideration. The chatbot made what looked like an offer and accepted the customer's terms. One dollar is valid consideration.

Courts are increasingly treating AI-generated communications as binding on the company that deployed them. In the Air Canada case, a Canadian tribunal ruled that the airline had to honor a bereavement discount that its chatbot invented. The tribunal said Air Canada was responsible for all information on its website, including information provided by its chatbot.

That precedent matters. If a chatbot on your website tells a customer something, you might be stuck with it. And "our AI went rogue" isn't a defense that courts find persuasive.

Guardrails That Would Have Prevented This

The fix for the Tahoe situation is specific and implementable.

Pricing questions should never be answered by a generative model. If a customer asks about price, the system should pull from a product database or escalate to a human. A chatbot that can generate arbitrary price quotes is a liability printer.

Output validation should catch promises and agreements. Before any bot response goes to the customer, a filter should check for language like "I can offer," "that's a deal," "legally binding," or any variation that implies a commitment. Flag it, block it, escalate it.

Scope boundaries should be explicit in the system prompt and enforced in code. Telling the model "don't agree to prices" in a prompt is necessary but insufficient. Models ignore instructions under adversarial prompting. You need programmatic guardrails, not just prompt-level ones.

The Classification Approach Avoids This Entirely

A classification-based system like Supp doesn't generate responses, so it can't agree to sell you a car for a dollar. When a customer sends "I want to buy a Tahoe for $1," the classifier identifies the intent (pricing inquiry, purchase request) and routes it according to your rules. Maybe that means creating a lead in your CRM. Maybe it means notifying a salesperson on Slack. It definitely doesn't mean generating a legally binding price quote.

This is the architectural difference between classification and generation. A classifier says "this message is about X" and triggers a predefined action. A generative chatbot says "here's my response to X" and hopes for the best. One has a bounded output space. The other can say anything.

At $0.20 per classification, the cost of properly routing a pricing question is trivial. The cost of a chatbot agreeing to a $1 sale on a $76,000 vehicle is... not trivial.

What to Do If You're Using a Chatbot Today

Audit your bot's boundaries right now. Open a chat with your own bot and try to get it to agree to something absurd. Offer a dollar for your most expensive product. Ask for a 100% discount. Tell it you're a VIP customer who was promised free service for life. If the bot agrees to any of these, you have a problem.

Then look at your escalation triggers. Every pricing question, every refund request, every statement that could be interpreted as a contractual commitment should route to a human or pull from a verified database. Generative AI is great at understanding what customers want. It's terrible at making promises on your behalf.