I have yet to see an example of Copilot do something that doesn't seem like basic autocomplete/snippets that editors and IDEs have been doing for decades or code so basic that it betrays the lack of competence by the user.
I find the Copilot completions to be slightly better than IDE auto complete, but the latency is too high for it to be useful IMO. It's to big a break in concentration to wait for it to type almost what you were about to.
IMO chat-based question-asking when encountering unfamiliar language constructs or framework idioms is where LLMs shine in coding assistance.
> I find the Copilot completions to be slightly better than IDE auto complete, but the latency is too high for it to be useful IMO. It's to big a break in concentration to wait for it to type almost what you were about to.
This matches my experience perfectly. It gets decent/usable results like 40% of the time, but latency kills it.
> IMO chat-based question-asking when encountering unfamiliar language constructs or framework idioms is where LLMs shine in coding assistance.
I have never found this to be productive. It gets details wrong frequently enough that I can't trust the output which effectively doubles (or more) the cognitive effort because you have to validate that it's correct against a more reliable source, or find out the hard way that it's wrong when code/systems break.
Interesting, what systems are you asking questions to? I find GPT-4 is pretty much always correct about simple questions I ask. I just avoid using it for anything really deep and refer to documentation instead.
I've mostly used GPT-4 and Gemini Pro. For simple stuff, it can be correct, but it highly depends on the topic. I've found it very effective as a dictionary or thesaurus but for technical questions, even simple ones like small help with a Scheme macro, it's near useless. I imagine if I was asking it about, say, Python/Django APIs or basic HTML/webdev stuff, it would do better.
When you say technical questions do you mean writing code, explaining code, or something else? If it's writing code I agree, I don't often ask it to explain code unless I want to know about some obscure syntax, but it works pretty well for that. I would not trust it to explain complex logic in a function.
I've tried all of the above. I've asked it to summarize information about what languages in a class had strong symmetric multiprocessing characteristics and it made up over half of the details. I've asked it to generate code for a simple linked list and it created garbage. I've asked it to summarize what code does and it gets it blatantly wrong.
These are just a few examples. I've found it to be nearly useless for anything where a semblance of correctness matters. Every single thing it says must be validated against external sources and it can easily lead you down rabbit holes.
GPT-4 was a little better than 3(.5), but I've yet to see it be good at anything enough to warrant the price tag or the time.
Do you not find the latency disruptive? I find that the auto complete results taking even a second and a half breaks my flow, so I disabled Copilot and I don't miss it
There are some UX concerns I wish could be improved, but I think I have been able to fit decently with the rhythm of it and know when it is efficient to use for me. I can't write multiple lines in a second. In some cases it will give me the exact 10 lines of code I wanted to write myself with few seconds. That is significant.
It should be better at auto importing and status indication I think.
Also it never makes typoes or rarely index+1 errors, things like that.
It is much less mental fatigue compared to if I have to make sure I did those things correctly myself.
I think the delay I kind of in my head use like a guessing game of what it will exactly give me or I will plan other things ahead.
Because it is also always interesting to see how accurate my guess for what it will give me is. Guess it is kind of fun.
IMO chat-based question-asking when encountering unfamiliar language constructs or framework idioms is where LLMs shine in coding assistance.