This is why I stopped using copilot. It was very good at spitting out code that looked correct enough that I didn’t spot the mistakes. And they were mistakes I’d never make myself if I wrote it manually.
Maybe don't use it for huge chunks > 10 lines? The point of copilot to me is avg typing speed is 40wpm and read is 250wpm. A 5.25x speedup considering even after I type code I still read it over. There will be errors in copilots code sometimes so even if you said its only 50% accurate thats still a 2.5x speedup on the boilerplate it's used for.
You have to think about it like auto complete. You don't blindly accept what auto complete tells you, your code would never compile otherwise, you read what it proposes to you and you modify what you don't like.
I get that, but I meant how often does it introduce bugs in the situation the OP mentioned, using it for "a language or framework you don't usually use", where I won't know if it looks right or not because I'm unfamiliar. Are they going back to fix the code often?
Personally, I don't even rely on normal autocomplete very much when I code because I feel like if I don't know what I'm trying to call, then I don't know the language or library well enough, and it's best to go look at the documentation.
I recently used Copilot while learning Rust and it was immensely helpful for the learning process. There were a lot of language features that I learned about by seeing Copilot use them.
As for mistakes, I've got a good enough sense for reading code in any language to know if something is obviously wrong with an algorithm. The mistakes Copilot makes aren't usually to do with language-specific syntax, they're typically algorithmic and therefore easy to spot in any language. This makes sense if you think about GPT-3's output for English: it tends to be syntactically correct, but often completely wrong about facts.
It helps that Rust's compiler is so picky and so helpful, because on the rare occasions when Copilot was completely wrong about the language the compiler could set me straight pretty quickly. Writing automated tests also helped and was made easy by Copilot. I could write a single test case showing how to use the API, and then just write function names to generate the rest of the tests (obviously I would sanity-check the assertions Copilot produced).
I'm not sure, I use it A LOT for cryptographic code in Rust and I don't blindly rely on its output. I believe if there's a bug that was introduced it would be my fault with high probability, not Copilot's.
Auto-complete generally completes a single identifier, and the possible completion list is usually restricted by the types involved. So the likelihood of it getting a wrong one is lower, but even if it does, it's easy to spot.
I've simply never had an issue with this sort of thing in months of using co-pilot to be my fancy autocomplete. And if there was an issue I'd notice quickly enough, because I am still the primary pilot.
It is also great as a starting point for error messages, comments, etc.
I think the key is to still give a shit about the code you're writing.
My rule using copilot is to use it as a fancy typing assistant which will frequently make mistakes - so you need to read everything it writes for you, and have solid test coverage too in order to avoid accidentally accepting a dumb mistake.
And much more enjoyable. I haven’t tried Copilot yet, but the process of letting it generate something and then checking it for accuracy sounds mind-numbing and stressful. Kind of like supervising a car that’s on assisted cruise: I don’t enjoy that either.
Do you use it to write anything you want - even works on READMEs and comments! But like any good test - test it in a different way to how to implement it.