[az09mugen]

"The study said 86.66% of the generated software systems were "executed flawlessly.""

What's that susupposed to mean ?

Unit tests were written to have this percentage ? By human or AI itself ?

86.66% of how many LOC ?

How long will it take for a human (because AI can badly do) to debug the code ?

What was the purpose of generated code ?

Lack of useful context here. Maybe I missed information fast-reading the article.

IMHO, it looks like just another rant on how "good" LLMs can "write code".

EDIT : Sorry OP, I didn't see the arxiv link.

I'm on mobile and read a 10 Mo pdf isn't worth it. I'll try to read it on a computer though, looks interesting.

[az09mugen]

I don't want to undermine the work of the researchers because they did a good one with a solid protocol.

But the problem to me is the specification, basically one of the tasks was : "design a basic Gomoku game". Which implies a set of pre-existing rules, something already known and well documented on the web. For example on github you can find 1.2k repos in python with the tag 'Gomoku'.

In a company you will be less likely asked to code a Gomoku from scratch, maybe for an interview, at most.

So in my POV, it's just code synthesis from already existing stuff. And reviewed.

I think my job is safe.