[dgroshev]

It's a really bad TA that has no self awareness of its limitations. You are well aware that it hallucinates, right? You probably have observed it yourself in domains you're familiar with. But how would you recognise a hallucination in something that you have no familiarity with?

"Learning something new" from LLMs is like trying to learn something new from a con artist. Sure, they might actually tell you the truth, but having little knowledge of something is exactly the worst possible state to be in while trying to figure out what's true and what's hallucinated.

[netruk44]

When it comes to programming, when an LLM hallucinates something in an area I'm not familiar with, it's usually very obvious that it is not correct because it just does not work at all. For me, programming hallucinations have never fallen into the case where it appears like everything is working but actually isn't doing the correct thing.

So then I just forward that back to the model, or in the very worst case use my own brain to fix the problem the LLM generated.

In either case, I'm still better off than had the LLM not existed, because even an incorrect solution is closer to the correct solution than nothing at all.

I now have something that's close to solving my problem that I wouldn't have been able to come up with solely on my own. It just needs a little tweaking to be correct, which I can muster my way through with a little help from the LLM or Google.

[dgroshev]

You are talking about most obvious hallucinations such as inventing a non-existent API.

However, those are indeed trivial. What's not trivial is not using APIs that exist for your particular problem; using obsolete patterns; not recognising that a particular approach is a dead end; not recognising potential future problems. All those problems fall in the same bucket of not knowing what you don't know, yet having a (seemingly fine) solution, which is worse than not having a solution, because that would force you to go and figure things out yourself (presumably reading recent documentation, etc).

I recognise that some people approached programming in the same way before LLMs, copying stuff from SO and adjusting till it works without ever understanding what they're doing. I guess LLMs can be seen as a neat extension of that "learning style" (which btw is precisely what OP warns against), but it does seem somewhat dysfunctional.

[netruk44]

I don't think copying and pasting from stack overflow then fudging things until it works is always quite the same thing as using an LLM to help write code.

I've just spent the month of August learning to use Godot 4.0 with C#. Resources for this topic on the internet are scarce, Google only wants to show me the official Godot documentation and YouTube videos (which I don't watch because I don't like to learn in this way).

Sure, I could spend my time reading the official docs cover to cover, but I much prefer to get my hands dirty. I'm not a novice game programmer, I've written my own game engines, used Unity and published games. I've got the underlying knowledge of how Godot might work, but I'm unclear as to how to accomplish specific goals I'm trying to achieve.

And absolutely, I could just put everything into Google, and probably find correct documentation for what I need. Or I could just write "// Interpolate from current position to camera" and have Copilot spit out "transform3d.interpolate_with" along with supporting code specific to the code I have right here, which also references functions I don't even know existed. This way, I'm learning the "API landscape" in a sense.

Godot 4.0 is new enough (and C# used with Godot used so infrequently in online resoures) to the point where Copilot literally can not produce the correct output basically ever. The API changed and everything it outputs is subtly incorrect, either generating code that is now obsolete, or that the code it's generating no longer functions because of the API change.

When this happens, I just dive into the docs as a backup strategy. But never as my first choice.

Is this worse than not having Copilot? I don't think so, but it seems like we probably won't agree on that. I still feel like I've 'learned' Godot (I at least know more now than I did on August 1st). I'm able to synthesize the code more easily on my own thanks to the starting points provided by the LLM. And I'm also writing tutorials on how to use Godot, further solidifying the things the LLM showed me in my own mind.

I'm not saying LLM's are a "one stop shop" for learning. I'm just saying it's another tool. One that I think is a little shortsighted to just completely disregard because it's sometimes incorrect.

[mhdi_kr99]

I'm on board with your point here

[jebarker]

My experience has been that the hallucinations in GPT4 are actually pretty rare. But in any case, if I choose to use code it suggests I ask it for explanations and then I verify those myself by other means, e.g. tests. I think it's too strong to call it a really bad TA. I'd say it's an imperfect TA and you need to check it's work, but it's work still has great value.

[dgroshev]

It's not just things that can be caught with tests. You wouldn't know if you got recommended an obsolete API, a dated pattern, or a charged foot gun, and if you're looking up docs for every suggestion I doubt that'd save you much time compared to reading docs through to begin with.

[jebarker]

I can definitely see those things being problems, but they are much more likely to occur when asking an LLM to generate code for you. That's not the best coding use case for LLMs IMO, it's much better to provide it code already written and ask it for explanations, bug fixes, refactoring etc. Then it rarely introduces new API calls etc

[skybrian]

No, it's more like getting bad advice from Stack Overflow or Reddit.

Usually you notice it's broken because it doesn't work.

There may be some advice that seems to work, but has subtle issues. (For example, race conditions.) Having more experience can help in spotting these problems.

Either way, copying code blindly isn't a good idea. In a professional situation, better testing and code review can help.

[dgroshev]

Yes, but collectively we understand that SO or Reddit are unreliable, plus it's usually one-way and we don't see people posting bad advice defending it just for us. LLMs, on the other hand, are extremely convincing, while being even less trustworthy than SO/Reddit.

Just make a mental experiment and imagine that every comment here praising LLM coding skills is actually about copypasting stuff from SO. Does it not seem alarming to you?

[skybrian]

I don't think it's any more alarming than people posting that they find Stack Overflow useful sometimes. That's why it became popular, right?

Similarly for searching the Internet. Internet searches are potentially dangerous if you can't tell good answers from bad. But if you treat it as a source of hints that you need to verify yourself, it's a pretty good source of hints.

Many of us have learned quite well that LLM's are unreliable. Someone posts about the problem whenever it comes up. It's something close to conventional wisdom.

It's still surprising the first time you see it hallucinate, though. You have to see it for yourself. Once you've seen it a few times, it isn't so convincing anymore.