[jackbucks]

It was definitely an interesting way to allocate pointers. I did once have a very large project where devs didnt understand this and resolved hundreds or more off by one and memory overwrites in C due to this feature.

But at the same time, I think blaming the software was kind of a cop out. Devs were in a hurry and simply didnt respect the rules. Given todays software engineer at large. Nerfing programming languages so they cant destroy things might not be a bad idea. But AI will nerf everything.

[fragmede]

why is AI gonna nerf everything? sure it could be used as the easy button, but I just spent two hours this morning learning about the neuroscience of how memory works in the brain that I didn't mean to and now I want to run studies on how memory works.

Why do you assume that AI is gonna nerf everything?

[AnimalMuppet]

AGI might. AI? No way.

See, AI was trained on existing data - on all that existing C code out there (sure, and also on all the papers and articles saying what was wrong with that C code). Those bugs are in the training data, and often not marked as bugs. So when AI generates C code, is it going to avoid making the mistakes that human code made? No, it's going to generate the kind of code it was trained on. How could it be otherwise?

That's not going to nerf anything.

[deathanatos]

> See, AI was trained on existing data - on all that existing C code out there (sure, and also on all the papers and articles saying what was wrong with that C code). Those bugs are in the training data, and often not marked as bugs. So when AI generates C code, is it going to avoid making the mistakes that human code made? No, it's going to generate the kind of code it was trained on. How could it be otherwise?

The generalization of this is why I think all these AI companies writing blog posts where the marketing department is just jer—ranting endlessly about how AI will improve itself into the singularity is just crazy talk. They generate a random statistically likely output, and the most statistically likely output is mid. Exceptional outputs — the ones that wow us or move the needle are exactly that, unlikely. AGI is sci-fi, and LLMs will not change that.

You can see the same effect when AI emits bash, too, and especially so since most bash is terrible, and most users of bash do not put in the effort to learn bash and its foibles. So it outputs what most people write, which is not great.

[AnimalMuppet]

It still could happen, if they had a way to judge the exceptional outputs from the mid and terrible ones. But I'm not sure they have that...

[CamperBob2]

When's the last time you saw a decent coding model create a buffer-overflow bug while trying to use C strings?

Serious question. Anyone else seen this happen in the last 12-18 months? If so, which model and version were you using?

[macintux]

Would you even know? Serious question. The volume of code the models can produce, the subtle ways these bugs can manifest (or even only manifest when under attack), it seems like they would be easy to overlook.