[araghuvanshi]

Well LLMs are claimed to be good at math too, and yet they can't count. Same point with the long contexts. And our actual use case (insurance) does need it to do both.

My hope from this article is to help non-AI experts figure out when they need to design around a flaw versus believe what's marketed.

[famouswaffles]

>Well LLMs are claimed to be good at math too, and yet they can't count.

You're putting a lot of weight into counting. I don't know anyone who wants to use a LLM after hearing "good at math" for counting of all things. Algebra, Calculus, Statistics, hell I used Claude 3 for Special Relativity. Those are the things people will care about when you say math, not counting.

Look, just test your use case and report that lol.

[araghuvanshi]

Look man, Claude 3, GPT4 etc didn't work for my startup out of the box. I thought it would be helpful to tell others what I went through. Why hate on the truth?

[famouswaffles]

Test the LLM on what you want it to do not what you think it should be able to do before what you want it to do. It's not hard to understand here and I'm not the only one telling you this.

Your article would have been very helpful if you'd simply did that but you didn't so it's not.

[stevenhuang]

But LLMs are good at math, they just aren't good at arithmetic.

https://www.lesswrong.com/posts/qy5dF7bQcFjSKaW58/bad-at-ari...