[kiwih]

Yes, the work definitely lends itself towards the question "is this better or worse than an equivalent human developer?" This is quite a difficult question to answer, although I agree that simply giving a large number of humans the same prompts could be insightful. However, then you would be rating against an aggregate of humans, rather than an individual (i.e. this is "the" copilot). Also, knowing research, you would really be comparing against a random corpus of student answers, as it is usually students that would be participating in a study such as this.

Nonetheless, we think that simply having a quantification of Copilot's outputs is useful, as it can definitely provide an indicator of how risky it might be to provide the tool to an inexperienced developer that might be tempted to accept every suggestion.

[laumars]

Rather than comparing against students in lab conditions, I’d be more interested to see it compare to students with access to Stack Overflow et al vs students with access to just Co Pilot. Ie is a junior developer more likely to trust bad suggestions online vs bad suggestions made by Co Pilot?

[sdevonoes]

Junior engineers will trust whatever information is provided to them as long as it is easily accessible. The reason juniors consult Stack Overflow is because is one Google search and one click away, whereas consulting the official documentation/reference takes more effort (because they usually don't appear on Google when one searches for errors/bugs/how-to). If Copilot (or another similar tool) is very well integrated in whatever IDE a junior is using, you can be sure it will be used and trusted because it will be faster than Google+SO.

[laumars]

That’s a very uncharitable view. My view point was a lot more around whether the extra information provided in SO (and others) helps an engineer to make better informed decisions about code quality and how many engineers will simply take the top answer verbatim without testing (either online or via Co Pilot).

The reason I singled out juniors has nothing to do with them being the least likely to check documentation. In fact seniors are just as bad in that regard. Plus a lot of the time the reason an engineer goes to SO is a particular common problem doesn’t have a pre-built solution in a languages standard library. The reason I suggested juniors is just because that’s who the researchers said they mostly work with.

So let’s be clear about one thing: I know plenty of junior developers who have put plenty of seniors to shame. I’m not about to suggest that juniors are worse developers; maybe less experienced in the chronological sense but you need to be damn careful before making other generalisations.

[sdevonoes]

> So let’s be clear about one thing: I know plenty of junior developers who have put plenty of seniors to shame.

Agree. My comment was more about: most of the engineers out there (and yes, I do include myself) tend to rely and trust on information that is easily accessible; no matter if they juniors or seniors. I think this also applies to everyday life in general as well (e.g., we tend to read the newspapers to get "informed", but we rarely go to the source of truth to check our "facts").