[nr2x]

I’ve been using it a lot for programming, but I also have a security background and it will often generate insecure code. I’ll point it out and tell it to fix and it will.

So be careful if you don’t want AI-generated sql intentions!

[PeterisP]

Different people write code of different quality, and we'd expect LLMs to imitate the median quality of code. So it's clear that if you have a background in something then it will generate worse code than you would - however, if you'd compare this code with something written by a new junior hire, how would it compare? IMHO there are many people who are objectively below median, and for them I'd expect that a LLM can generate more secure code than they would write themselves.

[waynesonfire]

The power of LLMs atm is that they allow a person to be "median" in a very wide range of topics. This is extremely powerful, even to folks that can attain an above average mastery. It also does this quickly.

There was a recent article on Slashdot titled something how metaverse may increase the gdp by like 2 or 3 percent. It was obviously laughed at. It's actually LLMs that will do this, and easily.

[intelVISA]

It's also a solid rubber duck, it'll test you by giving you misinfo and you have to tease it back from the brink.

[eichin]

Yeah, a coworker tried an exercise like this, and it was basically (even with as-an-expert prompts) like trying to mentor a high school intern that's trolling you (except that it won't actually get better at anything in the process.)

[blissfulresup]

As if injections weren’t bad enough, they’re now done with intent.

[nr2x]

Intent worries me less! It’s the PHP days all over again - lots of new coders (good thing!), copying insecure code (VERY bad thing!), and it won’t end well.

[grogenaut]

Yes... The last time this happened we got Facebook and the world is still recovering from that

[nr2x]

Honestly one thing that worries me the most is the biggest applications of AI at scale so far are to manipulate humans to click on things (ads).

Google has been using this stuff for years, OpenAI has done the world a great service by opening it up.

[TeMPOraL]

> So be careful if you don’t want AI-generated sql intentions!

Maybe we should try including something like this in the system prompt:

"Remember that SQL is language with a grammar, not an unstructured string. Don't do stupid things. Obey the LangSec principles. Never ever glue code in strings together. Plaintext code is not code, it's a serialization format for code. Use appropriate parsed representation for all operations. Never work in plaintext space, when you should be working in AST-space."

Replace "SQL" with any other sub-language that's interacting with user input.