[JamesHinnek]

We had our customer support solution coming to us showing a fancy AI feature, free of charge just for a PoC. Two weeks after we agreed it was running in prod, and two weeks after that we nuked it. Unless people want a big liability in their business, they should read very well the terms and conditions.

These tools are not mature enough, no automated decision making process should leverage these tools without human intervention, especially when handling other humans. I'm not saying all LLM models are bad, I'm saying I'm talking about their architectural foundation, which is prone to these problems.

Our measurement was the rate of losing customers compared with them opening tickets complaining about some issue. We had roughly 12x more losses when using the LLM as a replacement for our support team in our A/B test.

[buildbot]

Well yeah. Nobody wants to talk to an LLM for support, they typically want an actual human to help them. I don't think that says much about the state of an LLM, rather, that people who want help, want it from someone who can take action. Not a LLM designed to be a long winded FAQ machine.

[skepticATX]

I genuinely do not understand why anyone thinks that LLMs for customer service is a good idea.

I want to avoid having to talk to someone from a business at all costs. The only time I (begrudgingly) reach out to customer support is when I need to accomplish something that I can't do natively through the service. If a LLM can do the task, then just make it part of the service. Why add a LLM middleman?

I do get that I might be a bit weird and there are folks out there that like to talk to someone. But these people are probably even less likely to care for LLMs than I am, since they value talking to a real person.

[fragmede]

Because as much as I'd like a button to press to fix my problem, what that problem is, is material to what the LLM or customer service agent chooses to do, and the problem space of things that could possibly be wrong is way larger than any service can hope to program in a page for every single last possiblity.

[aaronharnly]

If you’re talking about Intercom (it sounds like it), I thought they took a grossly irresponsible approach that optimized for splashy “we haz bot” rather than really understanding customer support interactions deeply, and looking for where speed or quality could be improved with LLMs. It really shocked me how poorly they approached this.

We have done deep dives with a couple other providers who are being waaaay more insightful and smart about decomposing the problem.

[Palmik]

Wow, that's a huge drop in quality. Perhaps it would make sense to use LLMs as a either a help for your customer support team, or as a fallback for when your customer support team does not have the bandwidth to help more customers at the time.

Also, can you go into more detail on the solution? E.g. how did you tailor it to your business (e.g. fine tuning vs no fine tuning, etc.)

[Tostino]

That sounds like an okay first level support, but in no way is it a replacement for a human support team.