[Veuxdo]

> I expect it to get me in the ballpark faster than I would have on my own.

This is great if you are an experienced developer who can tell the difference between "in the ballpark" and fixable and "in the ballpark" but hopeless.

[panarky]

If 52% of responses have a flaw somewhere, then 48% of responses are flawless.

That is amazing and important.

The headline should be "LLM gives flawless responses to 48% of coding questions."

[thebytefairy]

There are articles every day about how AI is replacing programmers, coding is dead etc, including from the nvidia ceo this week. This kind of thing shows we are not quite there yet. There are lots of folks on twitter etc who rave about how genAI built a full app for them, but in my experience that comes with a huge amount of human trial and error and understanding to know what needs to be tweaked.

[kcplate]

> This kind of thing shows we are not quite there yet

I think you need to probably consider the time it took to go from 100% wrong, to 90, to 80, etc…. My guess is that interval is probably shrinking from milestone to milestone. This causes me to suspect that folks starting SWE careers in 2024 will not likely not be SWEs before 2030.

That’s why I tell my grandkids they should consider plumbing and HVAC trades instead of college. My bet is within 10 years nearly every vocation that you require a college degree will be made partially or completely obsolete by AI.

[panarky]

I tell my grandkids that vocational school is a perfectly decent and honorable way to get into a trade that pays better than retail.

I also tell them that a good university is a perfectly decent and honorable way to begin a life of the mind. It's not the only way, and a life of the mind isn't the life everyone wants.

I also tell them that the purpose of a university education is mostly not about training for a job.

[gwill]

something could be flawed but still arguably correct. the headline is fine. i wouldn't buy a calculator or read documentation that is 48% correct.

[nkozyra]

> This is great if you are an experienced developer who can tell the difference between "in the ballpark" and fixable and "in the ballpark" but hopeless.

While true, Stack Overflow wasn't much different. New devs would go there, grab a chunk of code and move on with their day. The canonical example being the PHP SQL injection advise shared there for more than a decade.

[cesarb]

> While true, Stack Overflow wasn't much different.

StackOverflow has the advantage that, if an answer is wrong, other users can downvote it, and/or leave a comment explaining the mistake.