[estebank]

An issue is immediacy: problems are better the earlier they are pointed out (why online errors are better than compile errorswl, which are better than CI errors, which are runtime errors). Having to copy paste an error adds a layer of indirection that gets in the way of the flow.

Another is reproducibility and accuracy: LLMs have a tendency to confidently state things that are wrong, and to say different things to different people, the compiler has the advantage of being deterministic and generally have better understanding of what's going on to produce correct suggestions (although we still have cases of incorrect assumptions producing invalid suggestions, I believe we have a good track record there).

If those tools help you, more power to you, but I fear their use by inexperienced rustaceans being misled (an expert can identify when the bot is wrong, a novice might just end up questioning their sanity).

Side note: the more I write the more I realize that the same concerns I have with LLMs also apply to the compiler in some way and am trying to bridge that cognitive dissonance. I'm guessing that the reproducibility argument, ensuring the same good error triggers for everyone that makes the same mistake and the lack of human curation, are the thing that makes me uneasy about LLMs for teaching languages.

[kolbe]

FYI, in VS Code, you highlight the error in the terminal, right click and select "copilot explain this." One less layer of indirection. In C++, I ultimately only end up using it for 10% of the errors, but because it's the type of error with a terrible message, copilot sees through it and puts it in plain English.

I was so impressed with gpt-4's ability to diagnose and correct errors that i made this app to catch python runtime errors, and automatically make gpt-4 code inject the correction: https://github.com/matthewkolbe/OpenAIError

[tialaramex]

Certainly for the only new diagnostic I wrote for Rust, I expect an LLM's hallucinations are likely to have undesirable consequences. When you write 'X' where we need a u8, my diagnostic says you can write b'X' which is likely what you meant, but the diagnostic deliberately won't do this if you wrote '€' or '£' or numerous other symbols that aren't ASCII - because b'€' is an error too, so we didn't help you if we advised you to write that, you need to figure out what you actually meant. I would expect some LLMs to suggest b'€' there anyway.