[feznyng]

I think the logic behind faster inference is that the LLM is unlikely to get it right the first time regardless of its intelligence simply due to the inherent ambiguity of human language. The faster it spits out a semi-functional bit of code the faster the iteration loop and the faster the user gets what they want.

Plus if you’re dealing with things like syntax errors, a really really fast llm + interpreter could report and fix the error in less than a minute with no user input.

[jmathai]

Also building something in this space. I think it’s a mistake to compare the speed of LLMs to humans. People don’t like to sit and wait. The more context you can give the better but at some point (>30 seconds) people grow tired of waiting.

[mritchie712]

yes, people are used to clicks / actions taking < 200ms, when something takes 20s+, it feels broken even if the results are good.