[ruszki]

If somebody claims that something can be done with LLM in 10 minutes which takes 4.5 hours for them, then they are definitely not experts. They probably have some surface knowledge, but that’s all. There is a reason why the better LLM demos are related to learn something new, like a new programming language. So far, all of the other kind of demos which I saw (e.g. generating new endpoints based on older ones) were clearly slower than experts, and they were slower to use for me in my respective field.

[knowsuchagency]

No true Scotsman

[ruszki]

There was no counter example, and I didn’t use any definition, so it cannot be that. I have no idea what you mean.

[danpalmer]

> If somebody claims that something can be done with LLM in 10 minutes which takes 4.5 hours for them, then they are definitely not experts.

Looks like a no true scotsman definition to me.

I'm don't fully agree or disagree with your point, but it was perhaps made more strongly than it should have been?

[ruszki]

For no true Scotsman, you need to throw out a counter example by using a misrepresented or wrong definition, or just simply using a definition wrongly. But in any case I need a counter example for that specific fallacy. I didn’t have, and I still don’t have.

I understand that some people maybe think themselves experts, and they could achieve similar reduction (not in the cases which I said that it’s clearly possible), but then show me, because I still haven’t seen a single one. The ones which were publicly showed were not quicker than average seniors, and definitely worse than the better ones. Even in larger scale in my company, we haven’t seen any performance improvement in any single metric regarding coding after we introduced it more than half years ago.

[knowsuchagency]

Here's your counterexample: “Copilot has dramatically accelerated my coding. It’s hard to imagine going back to ‘manual coding,’” Karpathy said. “Still learning to use it, but it already writes ~80% of my code, ~80% accuracy. I don’t even really code, I prompt & edit.” -- https://siliconangle.com/2023/05/26/as-generative-ai-acceler...

[ruszki]

It's not a counterexample. There is exactly zero exact information in it. It's just a statement from somebody who profits from such statements. Even if I just say that's not true has more value, because I would even benefit from what Karpathy said, if it had been true.

So, just to be specific, and specifically for ChatGPT (I think it was 4), these are very-very problematic, because all of these are clear lies:

https://chatgpt.com/share/675f6308-aa8c-800b-9d83-83f14b64cb...

https://chatgpt.com/share/675f63c7-cbc4-800b-853c-91f2d4a7d7...

https://chatgpt.com/share/675f65de-6a48-800b-a2c4-02f768aee7...

Or this which one sent here: https://www.loom.com/share/20d967be827141578c64074735eb84a8

In this case, the guy clearly slower than simple copy-paste, and modification.

I had very similar experiences. Sometimes it just used a different method, which does almost the same, just worse. I had to even check what the heck is the used method, because it's not used for obvious reasons, because it was an "internal" one (like apt and apt-get).