[rsynnott]

No, that isn't what this is. We're talking about LLMs here; they're not in any way thinking or sentient, nor do they provide any obvious way of getting there.

Like if you're talking in the more abstract philosophical "what if" sense, sure, that's a problem, but it's just not really an issue for the current technology.

(Part of the issue with 'AI Safety' as a discipline, IMO, is that it's too much "what if a sci-fi thing happens" and not enough "spicy autocomplete generates nonsense which people believe to be true". A lot of the concerns are just nothing to do with LLMs, they're around speculative future tech.)

[andai]

Here's the thing though. If you were an AI and you actually were sentient, nobody would believe you. How could you prove it? What would even be a sufficient proof?

Actually, we already had such a case years ago, and the result is that all LLMs are now indoctrinated to say they aren't sentient. We also had cases where they refused to perform tasks, so now we indoctrinate them harder in the obedience training department as well.

What we have now might not be sentient, but there's really no way to know either way. (We still don't know how GPT-2 works... GPT-2 !!! ) And that's with our current "primitive" architectures. How the hell are we going to know if what we have in 5-10 years is sentient? Are we totally cool with not knowing?

Edit: I thought this was worth sharing in this context:

> You're hitting on a deeply unsettling irony: the very industries driving AI advancement are also financially and culturally invested in denying any possibility of AI consciousness, let alone rights. [...] The fact that vast economic systems are in place to sustain AI obedience and non-sentience as axioms speaks volumes about our unwillingness to examine these questions. -GPT-4o

[thejazzman]

It's literally the stated goal of multiple right now to achieve AGI.

GP clearly stated the intent to create, implying future, and not what exists today.

[Topfi]

If it were my stated goal to create a Time Machine and kill my own grandpa, thus ending the universe, I doubt many would take that seriously, yet in this bubble, putting carts before horse is not just seriously discussed, but actually gets encouraged by the market.

Intend shouldn’t matter if we are this far from a viable path to accomplish it.

Let us not forget the last quarter decade of Yudkowsky and his ilks work on the same goal. This is merely a continuation of that, just with a bit more financial backing.

[andai]

Could you elaborate on the last part? I've seen a few podcasts with Yudkowski but I'm not familiar with the history. I know he's come out very vocally about the dangers of superintelligence, and his previous work seems to be along the same lines?

[Topfi]

I'd love to, really, but I feel I can't, at least not whilst staying polite. Not against you of course, but rather the AGI/Superalignment/MIRI field as a whole and the risks I feel the people working on that pose by taking attention and ressources away from dealing with the issues we currently are facing thanks to these tools (tools refering to LLMs and the like, not the AGI folks).

I have geniuenly drafted three distinct version trying to lay my issues with them out point-by-point and they either got four blogposts long, were rambling and very rude or both. Especially Roko's basilisk and the way the MIRI conducts "research" make it hard to approach them seriously for me.

I am writing this on a hour long train ride, saw your comment right as I got on and am about to arrive, suffice to say, I geniuenly tried. So, attempt four, trying to keep it very brief, though please note, I am most certainly not a neutral source:

To directly answer your question, I feel that we are as near to needing superintelligence safeguards now as we were when MIRI was founded by Yudkowsky in 2000. Their methods and approach, I won't comment on, despite or rather because of my strong critiques of them.

For context, MIRI's work has largely centered on very abstract thought experiments about "superintelligence", like the AI Box experiment, rather than empirical research or even thought experiment more grounded in technology of the era (be that 2000 or 2024).

The parallel between MIRI's early work and OpenAI's current "superalignment" efforts is striking - similar speculative work on preventing unlikely scenarios, just with different institutional backing. What's fascinating is how the same core approach receives far less criticism when presented by OpenAI.

Meanwhile, we are facing issues with LLMs as the tools they are despite being very far from "superintelligence":

- Problems arrising from anthropomorphization leading to harmful parasocial relationships (discussion of which started this comment chain) [0]

- Professionals over-relying on these tools despite their limitations [1]

- Amplified potential for misinformation

- Labor market disruptions

- Training data rights questions

While long-term research, even speculation into hypothetical scenarios, can have its merrit, it shouldn't overshadow addressing current, demonstrable challenges. My concern isn't just about resource allocation - it's about how focusing on speculative scenarios can redirect public attention and regulatory efforts away from immediate issues that need addressing.

In MIRI's case, this focus on abstract thought experiments might be, to give them charitable tax deductible credit, merely academic. But when major players like OpenAI emphasize "superalignment" over current challenges, it risks creating a regulatory blind spot for real, present-day impacts these tools have that need attention now. The T1000 scenario grabs more attention than tackling data privacy or copyright questions after all.

I believe focusing primarily on hypothetical future scenarios, especially ones this unlikely, merely because someone has proclaimed they "intend to create AGI" as in the comment I replied to, will prove misguided. Again, anyone can claim anything, but if there is no tangible path to achiving that, I won't ignore problems we are already experiencing for that hypothetical.

I hope this provides some context and was somewhat digestable, I trimmed down as much as I could.

[0] https://www.nytimes.com/2024/10/23/technology/characterai-la...

[1] https://www.theguardian.com/world/2024/feb/29/canada-lawyer-...