[fakwandi_priv]

When I'm reading reviews of plans created by an agent especially on security boundaries it's suggesting huge matrixes to test even the very obscure situations, but then I'm also reading things like this and I just don't understand. Are we even using the same tools?

[franga2000]

A large part is also how much you read back what the model writes. The good models generally write quite secure code, but they also often implement temporary solutions that they tell you to fix later.

For example, if secret storage methods aren't specified in the prompts, a model might decide to be clever and implement a generic secret access interface, with a default implementation that hardcodes everything. It will probably tell you that this is not production ready and you should write or specify your preferred secret storage implementation, but if you don't read or understand that, you'll just leave it as is and push to prod.

[shakna]

Management think models mean juniors can do senior work. Juniors don't know the footguns. Juniors can't read the code that the system outputs. Models get overwhelmed in any decent sized codebase.

Why would you be surprised there are failures?

[titularcomment]

1. It depends on model and tokens spent 2. Models talk the talk but not always walk the walk

[orbital-decay]

Tools are already preventing IoT companies from doing a ton of things they do, by default. It's a problem of the process, churn, and culture, not tools. I don't have any doubts that if given a coding agent that cares more than they do they'd still force it to hardcode a password or something because they feel like it's more convenient. Nobody cares there.