[SpicyLemonZest]

They say the second thing repeatedly and emphatically. You may not be aware of it because, when they do, critics make fun of them for believing a computer program could be so dangerous that the authors need to put controls on how it may be steered.

[sensanaty]

People "make fun of them" because they say they're building some uber-dangerous deity, yet take literally 0 steps to, I dunno, slow the fuck down for a bit?

Maybe people would take the threats more seriously if the hypemen weren't simultaneously claiming that we have to go at warp speed with all of this.

[Avicebron]

That's not why critics make fun of them. It's because their answer to "oh no we're accidentally creating the godhead. Someone please, give us power, your money, and praise, it's the only thing we can do."

It's vile hypocrisy. If they want to be priests, strip them of everything and they can live and work out of a concrete box in a mid-western cornfield. Why the material distraction if they are so religiously pure.

I know these people and I can tell you they aren't close to as smart as they think they are. Do you remember Yudowsky's "math petss"?

[simoncion]

This critic also makes fun of them because they go on and on and on about how vitally important it is to produce a safe tool that won't do harm, when their core products frequently consider attacker-controlled instructions to be its system instructions or its user's instructions, and are known to confuse their own internal chatter as instructions from their user.

Reliably differentiating between trusted, tainted, and untrusted data and ensuring that you don't mix the latter two groups in with the former is something we've known to do for nearly a half-century. Hell, even the youngest plausible programmer at the LLM companies is all but certain to be aware of SQL injections. And yet, despite their claims about being so serious about safety, they show zero interest in following long-proven software safety practice and rearchitecting their software to make it impossible to mix system, user, and attacker-controlled data. [0]

[0] One might argue that the fundamental nature of LLM-based systems makes this impossible. If that were true, then it would mean that these systems are impossible to make safe... the only safety option available would be to establish comprehensive blacklists, which is simply infeasible.

[pixl97]

LLMs are impossible to make safe in the same sense that humans cannot be made safe. There is no such thing as out of band data in the human mind.

For example, you have a dictatorship and need to track what the democratic countries are up to. The vast majority of citizens don't have access to information so will remain indoctrinated, but how can you be sure your data analysts will remain that way? You can't. So you take a batch out and shoot them at regular intervals.

The only winning move is not to play, but we're already past that point.

[Terr_]

> LLMs are impossible to make safe in the same sense that humans cannot be made safe.

I am very conflicted by this sentence, the two halves being:

1. Yes, the futility of making LLM's "safe" in that rigorous way is insurmountable, barring a major algorithm rewrite, and nobody really knows what that could be yet. Anyone who says it's easy is glossing over details--or selling something.

2. No, the failure modes of LLMs are substantially different than humans. If someone thinks they're similar, then they will fail at estimating and containing the risks. Now, perhaps if the comparison was to a brain-damaged human hopped up on psychedelic mind-altering drugs...

Note that I'm distinguishing here between the LLM itself--the hyper-mad-libs story generator--versus regular programs around it.

[simoncion]

> LLMs are impossible to make safe in the same sense that humans cannot be made safe.

No.

LLMs are impossible to make safe in the same sense that a car designed as if it was the ~1940's would be impossible to make safe for its passengers during an at-speed collision. There's only so much you can do if you're committed to using plate glass, rigid steel everything, and leaving out occupant safety belts because they're unpopular and spoil the lines of the cabin. [0] Back in the day, "the people in the cabin are the crumple zone" was state of the art, but we've learned an awful lot about how to make much, much safer personal vehicles in the ~75 years since then. It'd be massively irresponsible to design and sell a car today that ignored the safety and engineering lessons we've learned since then.

"Funnily" enough, the major LLM providers have designed and are selling access to systems that they very much want to be used in situations where you need a reliable, safe tool... but they've -somehow- ignored one of the most fundamental lessons we've learned about the design of safe software systems that are intended to be used in the presence of attacker-controlled inputs. [1] What they've done is no less irresponsible than designing and selling a new car that conforms to the very latest safety regs of the 1940's... AFAIK, it's so irresponsible to design and sell such a car commercially that -in the US- it's a violation of federal law to do so.

As an aside: you may have seen this video already, but it's worth a look if you have not. [2] Though, the classic car in this crash is equipped with safety glass, so -sadly- you don't get to see all that fun.

[0] One of my great-grandfathers spent the remainder of his years intermittently using tweezers to remove shards of plate glass migrating out of his face that had been lodged in there during an automobile accident that he was fortunate enough to survive.

[1] For more on this, read: <https://news.ycombinator.com/item?id=48999644>

[2] <https://www.youtube.com/watch?v=C_r5UJrxcck>

[Terr_]

I want to raise the possibility that you (pixl97, simoncion) actually hold many of the same opinions but are clashing because of a different definitions the LLM / bad-thingy scope.

* Narrowly - The core algorithm that extends documents cannot be made safe, because it's a stochastic machine with no data/instruction separation possible. Unanticipated input can evoke arbitrary output.

* Broadly - The overall offering (centered on the document-extender algorithm) could be made safe by limiting its over-ambitious scope, treating the document-extender output as malicious-by-default, and sharply limiting what that output can drive or influence. Of course, that would exclude the berjillion-dollar stock valuation replace-all-humans stuff.

[simoncion]

> ...but are using different boundaries for what constitutes "the LLM"

I'll make note that my original comment only used the term "LLM" in the phrases "the LLM companies" and "LLM-based systems". The latter use was in this footnote:

  One might argue that the fundamental nature of LLM-based systems makes this impossible. *If* that were true, then it would mean that these systems are *impossible* to make safe... the *only* safety option available would be to establish comprehensive blacklists, which is simply infeasible.

I acknowledge that my follow-on commentary -the one to which you replied- got sloppy with the terminology. I should have used the phrase "LLM-based systems", rather than "LLMs". I do feel that my original commentary was not at all sloppy with the terminology and made my position on the current state of the safety of the systems sold by the Big LLM Vendors and general understanding of where the bounds of the big pile of linear algebra and the bounds of the I/O to and from that pile lie clear.

[SpicyLemonZest]

Sorry, I don't understand this comment. Has Sam Altman ever said that you must praise him, or that he wants to be a priest, or that he's "religiously pure"? Unless I'm missing something, it seems like you're shadowboxing against a stereotype you've invented rather than the actual positions of AI research labs.