[kianN]

I see a fair number of comments here advocating for either codex to hand-roll this themselves, or to simply punt to SQL. I do want to advocate for the difficulty of the problem, even if I can't speak to the company itself.

At the scale of a few hundred to a few thousand documents, especially short documents, there are a few out of the box methods that can yield reasonable results, whether it be embedding clustering or leveraging LLMs for tagging.

However as your (1) datasets gets larger (2) documents expand from tweets and text messages to 30+ minute conversations and (3) you build downstream analytics on top of the learned semantic units, you really start to feel the limitations of LLMs and embedding for reliable annotation. That doesn't even get into the nuances associated with taxonomy management, seasonality, and model drift.

TLDR; this problem solved effectively has a lot of value and is a lot harder than it seems.

[bluelightning2k]

Why is it hard? Ultimately you take whatever your signal is and send it to some relatively cheap LLM.

How is it easier to sign up and manage a different service, implement a different API, etc.

And from the company side the fatal flaw is that these types of tools rely upon 1% of their users having huge spend. Nobody is going to be a huge spender here because it's easier to hand roll than navigate procurement on this (not to mention impossible to justify the spend, additional security/privacy risk, etc.)

It feels approximately impossible for this company to have large accounts.

[laalshaitaan]

it gets hard when you need this continuously across lots of chats/calls, with metadata, changing clusters, going deeper into a user journey, etc. the LLM call is just one part of it lol

we're keeping it useful every week, finding out insights that the teams can extract value out of, work with them to understand users better.

the procurement what we've seen is v similar to how one would have for any analytics product? and we're selling this to companies when/once it becomes someone's job to do this

[pqtyw]

If you get this to work once rerunning it weekly seems fairly easy unless you actually need to see the data live and have perfect uptime?

[laalshaitaan]

a lot of our customers want a daily morning report on slack & flag things instantly rather than to wait for a week so thats why we keep it realtime

[laalshaitaan]

yea, at our volume which we still consider small as we've been able to figure out a way with llms & embeddings, its still fine. + we onboarded a voice ai company with more than 2 hour calls and thats when it was super hard to solve since there were so many elements to consider.

model drifting is something a lot of folks do face after 5th/6th turn as per my understanding and it usually the median, how did you tackle it if you have yet?

also yea, thats why we went for a per customer taxonomy than a general one, yeilded better results + easier to improve upon.

[kianN]

To clarify, I wasn't criticizing your approach or product, more responding to the people dismissing the problem you are solving.

Regarding my experience, I have done a fair amount of work in the contact center space with long calls. I used statistical Bayesian approaches which I found to be much more resilient especially on long documents than embeddings/transformers. It also provided a joint modeling foundation for classification with much lower label requirements than BERT or traditional ML.

[laalshaitaan]

im hearing this for the first time and damn! i just told this to my cofounder/cto and he said hes gonna give this a shot in the coming days.

damn, i read bayesian in statistics like years ago, never thought itll come back this way

[kianN]

Happy to chat more in depth if more details would be helpful. I think my contact info is accessible from my HN profile.

[pqtyw]

IMHO this would make more sense if provided as part of a larger "platform" like Langfuse/Langsmith/etc. Otherwise you just end with a dozen SaaS products for highly specific use cases which might not scale that well.

Realistically do you also need to have this live with a fancy? i.e. a custom solution maybe even Jupiter notebooks initially might be sufficient. It's not like 100k messages is a very large dataset. It's not trivial to make a generic solution that fits every use cases (besides of basic customer chatbots) to get actual value for more agentic products.

[laalshaitaan]

i get the push, most teams come to us after they've done/tired of the claude running analysis thing manually and want a pro-active thing.

we're also targeting conversation first use cases and for them this serves as their everything custoemrs. we obv do not sell the fancy part, idts that sells anymore lol. lot of our queries come from our slack app/mcp.