Maybe as a small anecdote: I had a coworker ask an AI to write them a function to auth to aws using boto3 and environment variables. The LLM happily complied and gave them a 30 line snippet but it failed to point out that if they were to open boto3 docs it would tell them that it will already default to environment variables when it has no credentials and the whole function is just reinventing the wheel.
If you’re throwing the LLM at APIs you don’t know, how could you possibly verify it is using them properly?
The question is how fast and well dev unfamiliar with that API would achieve the goal alone compared to being assisted by LLM: he could set up snippet, run test, see what are the error, ask LLM to check why error could happen.
I don’t believe that the time a dev spends on prompting an LLM over reading an API reference saves a significant chunk of the TCO for the lines of code they write.
If you’re throwing the LLM at APIs you don’t know, how could you possibly verify it is using them properly?