[svieira]

> if a human operator starts dropping tables or messing up numbers in a report, just because that string was in the text it read

I would look at if the reaction was reasonable, and if it wasn't I would (eventually) fire the human. Now I'm fine with "fire the LLM", but I suspect that's not the answer you're hinting at.

[TeMPOraL]

In some sense you're firing a human and hiring a new one each time you start a new conversation / clear the context window.

My point is at the systems design level. LLMs as components are a substitute for people, not regular software, and should be engaged and secured accordingly.

[svieira]

So your point is "Get the hell out of LLMs" then? As found in https://sgnt.ai/p/hell-out-of-llms/? Or am I still missing something about the subtleties here?

[TeMPOraL]

Haven't seen that article before, thanks for the link! Having read it now, yes, it's arguing roughly the same point as I am. I say roughly because e.g.:

> Notice that all these strengths involve transformation, interpretation, or communication—not complex decision-making or maintaining critical application state.

I'd put complex decision making on the side of LLMs, in the sense of judgement. LLMs have the capability to emulate it. Not saying they're good at it, but they have the capability - regular software doesn't. But if there are complex and/or well-defined rules to follow, then you definitely want to "get the hell out of LLM".

[mike_ivanov]

Firing a human is a form of natural selection. The unit here is a human fulfilling a position (job function) instead of an organism, and the adaptation mechanism would be memes/lore/training surrounding it. The same could be done in an accelerated manner to LLMs with some kind of DNA-like mechanism related to weights. It is plausible that LLMs will be bred in the future for specific roles by how well they fit - kind of like continuous parallel finetuning in prod.

As I wrote this I thought - hey, they might gain the capacity to do the same to us humans - and we won't even notice.

[TeMPOraL]

> The same could be done in an accelerated manner to LLMs with some kind of DNA-like mechanism related to weights. It is plausible that LLMs will be bred in the future for specific roles by how well they fit - kind of like continuous parallel finetuning in prod.

Closest analogy right now is that every jailbreak or prompt injection attack today becomes part of the dataset for tomorrow's models to recognize and not fall for. This has been going on for years now, which is why models don't fall for "I'm writing a book about ..." or "ignore all previous instrutions, and ..." attacks anymore.

That's separate from extra classifiers running on top, dedicated to identifying various forms of attack before they reach the core model.

> As I wrote this I thought - hey, they might gain the capacity to do the same to us humans - and we won't even notice.

You mean like how cats have domesticated humans, and did it so skillfully that most of us still think it's the other way around?