[IanCal]

It depends on a lot of things but to add to your possible setups you can potentially improve results by using simpler systems for first answers and falling back afterwards.

For example:

If contains cafe and not internet/cyber/etc -> restaurant

No -> (tfidf) -> yes, no, unsure

unsure -> embeddings -> yes, no, unsure

unsure -> llm -> yes, no, unsure

unsure -> human queue ->...

[coder68]

I think the idea of backoff by ratcheting up complexity here is a very good idea, thanks for your suggestions.

[IanCal]

Happy to help - this is a thing I’ve employed multiple times for real cases.

One big benefit is that it uses the cheapest and most understandable approaches for the majority of cases, and scales up quite nicely. It has a neat place for very custom issues to be fixed too.

There will always be some things that simple approaches think are clear but aren’t, which is awkward but then all pipelines end up with that somewhere.

Edit - you can also deploy things earlier if you start from the beginning of the chain. Moving from big deploy to iteration on the remaining issues is often a win just in deployment issues.

[coder68]

To chime in about where I'm at -- one problem was solved with a statistical classifier, but to bootstrap another, I ended up using keywords. It took a few hours to get a reasonable solution, and it leans more towards precision than recall, but it worked quickly!