[taurath]

This is religious fervor folks, as AI 2027 was.

I grew up in evangelical christianity, and to them the end of the world is just around the corner, the same way it has been since I was a small child and likely will be when we are all gone. This isn't science. This isn't hypothesis experiment record results. This is very expensive astrology, shiny rock collecting, ritualistic meaning-making and self-justification.

Yall, with your incredible wealth and resources you could do real good in this world and make society better, healthier, better educated, and the whole world more equal, just, and reduce the desperation and suffering. Reject the false and self-serving narratives that empathy doesn't matter, that altruism isn't "effective". You can change a person's whole life in a moment.

[skrebbel]

It's nuts how well "Superintelligence: The Idea That Eats Smart People"[0] aged. That talk is a decade old by now and still hits just as hard as it did back then, despite the incredible advances made in AI in the meantime.

[0] https://idlewords.com/talks/superintelligence.htm

[taurath]

Yeah I think its possible that for many folks its the first time they're coming up on these concepts, and it troubles them in the same way that the concept of death troubles them (and me, to be clear!).

For me its as simple as watching how people talk, and seeing how in every single case whatever the next thing is, if you believe It, there is only ever justification of doubling down, doing more, going deeper, reducing any doubt. These are not scientists, they're business people and salespeople, and a few optimists having recently on paper solved all their worldly financial needs.

Even if one throws that aside, spending time exploring and building with the most state of the art LLMs is just as instructive. I'm watching the implementation - whats working is ML models trained on specific domains (not much different than 5+ years ago), and whats not working is a general model that humanity can let go to work on its own. Sit in front and observe ideas turn to the samey intellectual, high-syllable mush. Its productive, but not in any way that's promised.

[YeGoblynQueenne]

>> Even if one throws that aside, spending time exploring and building with the most state of the art LLMs is just as instructive. I'm watching the implementation - whats working is ML models trained on specific domains (not much different than 5+ years ago), and whats not working is a general model that humanity can let go to work on its own. Sit in front and observe ideas turn to the samey intellectual, high-syllable mush. Its productive, but not in any way that's promised.

Important point. LLMs were early on hailed as the first general-puprose AIs that can perform any task (remember "Sparks of AGI"?). Today they're increasingly promoted for specialised applications - coding, as a for instance.

[Leynos]

While there are some coding focused models (composer, for example), the majority of frontier models are pitched as general purpose. The coding harnesses for Claude and GPT are even being repurposed as general purpose knowledge work harnesses.

[YeGoblynQueenne]

No, you're right of course, but I have a feeling it's much easier to sell a system with a clear goal, like "this LLM generates code" or "this LLM solves math problems". Even if the underlying model is a general purpose one. I think there's always a question, when one has a product, of "what does it do?". "This thing does everything you want it to" is not a great way to sell something.

More to the point, even models marketed as general purpose are clearly trained on specific tasks. That is, the AI companies want to promote their systems as general purpose but they also want to make them good at specific tasks, because that supports their marketing story, that those are general-purpose systems that are so powerful they can even do maths and science.

Or maybe code, maths, and (maybe) science are just the things they find it easy to train their models on, for different reasons and in different ways. You can also get a feel about the things they have tried to do and failed, e.g. real-world autonomy isn't really working (or not working yet, who knows), so OpenAI is not trying to sell an embodied generalist agent say, that can clean your hose, do your laundry and teach your kids maths and science on the side.

I mean even generating text is a very narrow task, in the general sense, compared to all the things that humans can do (never mind other animals) it's just that we use text so much and for so many things that there's an endless list of applications for a text generator; as we have all found out.

[red75prime]

> real-world autonomy isn't really working

It works in limited ways (but in the realest-real world nevetheless). Waymo, Wayve, Baidu Apollo, Tesla and others seem to rely on VLAs, VLMs or transformer models in general to do autonomous driving.

[dmitriy_ko]

> whats working is ML models trained on specific domains (not much different than 5+ years ago), and whats not working is a general model that humanity can let go to work on its own.

As usual, AI skeptics are moving goal posts. Modern LLMs are on a completely different level in terms of how GENERAL they are vs anything pre-LLM. You can give it a completely novel puzzle and it will solve it. 5+ years ago you had to train NN to solve particular type of puzzle.

[TheOtherHobbes]

Did you actually read the text? OPs are calling that Plan D.

They're proposing an alternative, which is a global brake on frontier AI research to keep the basilisk in its jar until we work out what we're dealing with and how to handle it.

[layla5alive]

No, they're proposing a spying panopticon and state control of global resource distribution - specifically general purpose compute - including seizing and destroying GPUs. They're proposing a totalitarian global dictatorship controlling computing hardware and software.

Lest you think I'm being hyperbolic: https://ai-2040.com/supplements/covert-ai-projects

This is arsonists selling fire insurance.

[esafak]

A nasty global inspections regime just like we have for nuclear weapons, which are less dangerous than this. Oh no ...

[layla5alive]

Nuclear weapons are less dangerous than compute? Are you listening to yourself?

Humans employing this kind of thinking and desire to control other people is the genuine danger here.

[tim333]

I always thought it a bit pessimistic on the upsides like robot to build us palaces, immortality, conquer the galaxy type stuff along the lines of the "Grandiosity" bit in the essay. It's a shame we have amazing tech possibilities and most people are either saying either it's rubbish and won't work or it'll bring doom. Aside from being boring they are probably factually wrong.

Bit like saying early medical knowledge was rubbish or will doom us by summoning demons while really life expectancy at birth went from twenty something in Roman times to eighty odd now.

[beshrkayali]

This is hilariously true

> AI risk is string theory for computer programmers. It's fun to think about, interesting, and completely inaccessible to experiment given our current technology. You can build crystal palaces of thought, working from first principles, then climb up inside them and pull the ladder up behind you. People who can reach preposterous conclusions from a long chain of abstract reasoning, and feel confident in their truth, are the wrong people to be running a culture.

I understand how people running in the same scene fall into the echo chamber effect and get gulped into the cult, but why does everybody want to be a prophet?

[0xDEAFBEAD]

Theorizing about nuclear winter is somewhat similar, in the sense of being inaccessible to experiment. Does that mean we should disregard the possibility of nuclear winter?

[m4rtink]

As with using the Orion drive for launch and landing on Erth is possible, test nuclear winter is certainly possible. But as with the former, you won't be very popular among the survivors (if any).

[walrus01]

Being a prophet is probably great until you suddenly find yourself building a fortified compound in Waco, Texas and purchasing black market full auto machine guns.

[saltwatercowboy]

Thank you for contacting Gunmetal Ranch, a legitimate 501(c). If your call is related to the class-action "most dangerous game" settlement, please hold for a cowpoke.

[int3trap]

> but why does everybody want to be a prophet?

Its not everyone building crystal palaces in their mind, they're all building fortresses. And they can't be wrong in their fortress or it breaks their world view which they cannot accept.

[tim333]

AI is "completely inaccessible to experiment given our current technology"?

I think experiments have happened and even been mentioned on HN occasionally and effected RAM prices.

[angoragoats]

> AI is "completely inaccessible to experiment given our current technology"?

No, that’s not what the article/talk said. It said AI _risk_ is inaccessible (as the poster you’re replying to correctly includes in their quote). The point in the original talk is clear if you read it; that everyone coming up with their own pet theories for how superintelligent AI could destroy us as a species is building their theory on speculation on top of speculation.

Note that the talk is from 2016, before transformers and LLMs were invented. Though I would argue that the vast majority of it, including the parts we’re discussing here, are still completely valid.

[GCUMstlyHarmls]

Video link, as the link on the page is dead. https://www.youtube.com/watch?v=kErHiET5YPw

[casey2]

and upon rereading completely holds up technically we are still passing in massive data into simple networks giving no opportunity for introspection or recursive self improvement.

[0xDEAFBEAD]

>despite the incredible advances made in AI in the meantime

So the goalposts will be moved whenever necessary in that case?

No amount of incredible advances in AI will ever get skeptical HN commenters to take AI's implications seriously?

"The Gish gallops will continue until the nagging doubts have been silenced"

[trebaud]

Or you deeply suffer from normalcy bias. Not believing that AGI is possible is irrational and unscientific imo, the human brain exists, it is not made of magic, it can reproduced. It is as simple as that. We can debate about timelines, architectures that may lead us there, etc... You can even talk all day long about definitions of AGI, and people waste their time doing that. But saying that the human brain cognitive capabilities cannot be reproduced on other types of substrates is stupid at this point.

[davemp]

There is a vast gulf between theoretically possible and technologically feasible.

If you can’t provide a realistic path to achieve something, you’re asking people to believe in science fiction.

You could tell me that a rock’s molecules are comprised of protons, neutrons, and electrons. Blood is also entirely protons, neutrons, and electrons; so theoretically, one could rearrange stone into blood. But without an actual method to do so, it sounds like you’re telling me that you can squeeze blood from a stone.

> the human brain exists, it is not made of magic, it can reproduced

Yeah. It only takes 9 months and ~18 years of training…

> But saying that the human brain cognitive capabilities cannot be reproduced on other types of substrates is stupid at this point

Let’s be clear. Everyone is talking about silicon transistors here. That’s what we’ve got.

Digital computers have real limits. Sensors and other sources of training data have real limitations. It’s not clear that we can organize them in a way to reproduce organic brains.

[themgt]

What's strange to me about these comments is they're timeless. They could have been written in 2026 or 2016 or 1966.

Like, afaict, for many on HN going from ELIZA->Fable 5 just didn't cause any update to priors regarding this whole philosophical question. The argument against has remained unchanged. I don't see any point in arguing about it, I just find it very strange.

[defgeneric]

It's a form of denial. We're getting another "de-thronement of man" on the order of Copernicus and Darwin. Some get excited, others turn away in horror. Negation is the outward expression of the desire to keep human intelligence wrapped in its mystical veil.

One popular idea is that these systems will asymptotically approximate human intelligence because they're trained on mostly human-written texts. Not only is that untrue, it's also directly contradicted by our experience with previous RL-systems, where they seem to breeze right by human ability without even the slightest hiccup.

[disgruntledphd2]

> with previous RL-systems

Most human systems are much, much, much more complicated than most closed world games (which is where RL approaches have seen massive success, mostly through self-play).

Like LLMs are great, but I honestly can't see us getting actual general intelligence out of them.

[layla5alive]

They already possess general intelligence by many metrics. Sure they miss a few, but that's nitpicky and goalpost shifty - lots of humans make errors of all sorts as well, or incur brain damage limiting them in one or a few areas of intelligence - we don't then say they are not general intelligence anymore.

I think maybe you mean superintelligence, which is a more fair critique.

[iugtmkbdfil834]

<< One popular idea is that these systems will asymptotically approximate human intelligence because they're trained on mostly human-written texts.

Can you elaborate on this? I am clearly not aware of this line of thinking and the related contradiction.

[defgeneric]

Approximate as a limit and not surpass, where the hidden hope actually seems to be that they don't surpass. There are lots of variations floating around, from simple metaphors like "LLM as librarian that speaks to you," to even Sutton's remark that rather than being a case where we've taken the "bitter lesson" to heart, LLMs may be a yet another case where we're limited by it.

[beepbooptheory]

Unpack this a little bit. Why is it strange or interesting to you? What specific priors need to be updated for us here? What is the philosophical questions at play for you?

[themgt]

To meta-unpack a little bit ... it is strange to me that Fable is far more capable of discussing these questions than apparently 99% of humans. Along with being more capable at quite a lot else than most humans.

[notahacker]

It doesn't seem at all strange to me that a chatbot trained by true believers in an AI singularity and the importance of safety guardrails will give more satisfying answers to true believers in an AI singularity and the importance of safety guardrails than talking to humans who might ask questions they're not prepared to answer (or might say nasty sceptical things or just not seem interested)

As for "updating priors", that goes both ways. There's plenty more reason to think "hey, transformers and RLHF might actually make some killer products" but certainly no reason to think the few people who didn't realise that "GPT3 is too dangerous to release" and "all software engineers will be replaced within 6-12 months" were marketing rather than prophecy have some kind of special insight into how it's all going to pan out. Clock's ticking to the promised 2027 reckoning too...

[beepbooptheory]

OK.. when you discuss these things with your Fable, what topics come up? Can you articulate one of the questions? I am probably just another dumb human FYI, but just try it out and we will see if I can follow along.

[zdragnar]

Fable 5 doesn't represent anything new, other than scale and some refinement techniques, over the original LLMs. In the chase for AGI specifically, LLMs are a dead end, just like all the other AI technologies that died in the AI winter.

What priors should be updated?

[0xDEAFBEAD]

>Fable 5 doesn't represent anything new, other than scale and some refinement techniques, over the original LLMs.

Yet it is generating billions in revenue which Eliza did not.

Perhaps all we need is scale and some refinement techniques to eat a big fraction of the economy.

If unimpressive inputs lead to impressive outputs, that should make you more worried, not less.

[zdragnar]

The commercial viability is orthogonal to whether it achieves AGI, which is effectively what "reproducing the human brain" amounts to in this discussion.

[m4rtink]

And possibly trilions in running costs, not mentioning all the shady training data sources.

[dwaltrip]

Strong disagree. Fable is first model that actually feels smarter than me in certain non-trivial ways.

It can hold many complex and partially contradictory thoughts in its head at once, in a way that feels significantly superior to Opus (for example). And then can make reasonable syntheses across these.

In a couple rounds of back and forth, with relatively low effort (but strategic) prompting, it produces complex, accurate analyses in 5-10 minutes that would take me multiple hours of hard, very focused work.

I still need to remain tightly in the loop, providing frequent course correction, clarification, high level reframing, nudging, and grounding.

It incorporates my feedback incredibly well.

It’s honestly staggering. Fable has changed my assessment of the current trajectory more than any model since possibly gpt-4. Opus 4.5 of last year might be a close second.

———

My advice for anyone who wants to get more value out of these tools:

When a model does something idiotic, don’t throw your hands up in the air. Be curious. Try to turn it into a puzzle to be solved.

It know it’s hard sometimes, especially if you are drowning in slop from other people… or generated by yourself, heh.

It can be exhausting. I struggle with this also. I have thoughts on how to make it better. We shall see.

[layla5alive]

Human brain looks a lot like scale and minor refinement above dog brain. Or mouse brain.

[iugtmkbdfil834]

I just realized that you might really be onto something. I wonder now if it is just a function of our very human inability to let go of a known construct that has served us well until now or something else. I have my own opinions, but as strange as it sounds, this may be the HN equivalent of ok boomer moment.

[throw310822]

Jesus... This morning while I was drinking coffee and staring at the screen (it's Saturday) an agent did the equivalent of days of my work, reading code, understanding, hypothesizing, comparing, using tools, writing scripts, launching compilers and running tests, identifying problems and proposing solutions, and more. Only someone who hasn't spent a second reflecting about what it means to think and to be intelligent can claim that we miss a realistic path to intelligence. It's so damn clueless and stubborn and confidently wrong that it annoys me immensely, so sorry for the rant.

[davemp]

> clueless and stubborn and confidently wrong

Uhuh. I really shouldn’t be replying to this type of comment from a throwaway.

But the extremely powerful semantic search that we get from LLMs isn’t enough. I don’t think anyone is credibly arguing otherwise?

Agents already are a layer on top trying to bridge the gap. But they’re really just using LLMs as a heuristic to explore extremely NP problem spaces. The notable successes with agents so far are when we can provide them with a solid verifier and preferably additional context hints on the steps to take in the problem space. See the test oracle problem on where this gets us.

So forgive me if I think that it would be enough of a jump in computational complexity to remove those guard rails that it’s not feasible. But don’t say that I’m clueless, stubborn, or confidently wrong.

[27183]

> reading code, understanding, hypothesizing, comparing, ...identifying problems and proposing solutions, and more

Except it did none of those things, really, because that's not how it works. This might help, it's a good writeup: https://www.0xkato.xyz/how-llms-actually-work/

We know how these machines work, it's not mysterious, there's nothing "extra" happening.

[eihli]

Now show me a writeup that explains how the brain works so I can understand why the brain does those things.

> We know how these machines work, it's not mysterious, there's nothing "extra" happening.

It sounds like you're saying "We don't know how brains work, they're mysterious, there's something 'extra' happening", and using that as justification for why you're saying a computer, an AI, can't "understand".

I think most people on Hackernews now who would use the phrase "my AI worked overnight and hypothesized, compared, etc..." already know how an LLM works, and still chooses to use those words. So the issue isn't that they don't understand. It's that they understand and still use those words. So the disagreement is somewhere else.

[27183]

I'm not claiming there's something "extra" happening in brains. Merely that we just don't know how they work well enough to use that knowledge to do engineering. Neural nets are quite unlike brains, despite the unfortunate shared vocabulary.

OTOH we do know how neural nets work, and they definitely don't do "thinking" or "reasoning".

[sudb]

This feels like a semantic disagreement to me? If an LLM got to an acceptable end result code-wise, what would you call the process that took place to get it there?

[27183]

I'd call it what it is: a good enough stochastic search result extracted from the model's embedding space.

[0xDEAFBEAD]

Just a few comments ago you argued that we don't know how to build superintelligence. Now you're saying we know how the (unevenly superintelligent) Fable system works.

It doesn't seem like you're being consistent here. I'm concerned there might be some motivated cognition going on.

"What is true is already so. Owning up to it doesn't make it worse. Not being open about it doesn't make it go away. And because it's true, it is what is there to be interacted with. Anything untrue isn't there to be lived. People can stand what is true, for they are already enduring it."

[27183]

I'm not following the "Fable" stuff, what is that? [edit: ah it's a new language model from Anthropic.. yeah still don't see how it's relevant here]

[throw310822]

> We know how these machines work, it's not mysterious, there's nothing "extra" happening.

Lol. This is more telling about your implicit unscientific preconceptions that you wanted to reveal. Of course there isn't anything "extra". Where do you think intelligence comes from, some mysterious realm? It's physical, computational. The fact that at the bottom we produced it via matrix multiplication is irrelevant. Maybe humbling. You are denying a visible fact (a machine performs tasks that require flexible analytical and cognitive skills) precisely because there is no magic happening anywhere.

[27183]

> Where do you think intelligence comes from, some mysterious realm? It's physical, computational.

Well, no. I don't think it comes from some mysterious realm. I think that which is not physical does not exist [edit: and if you like I'll follow that one right down the rabbit hole--continuity and infinity are useful delusions]. But that eminently does not mean we know what intelligence is, let alone how to build one.

> The fact that at the bottom we produced it via matrix multiplication is irrelevant.

Huh? We don't even know what "it" is. How can you say you produced it?

> a machine performs tasks that require flexible analytical and cognitive skills

You see that, I see a lucky stochastic search result. Don't underestimate the "creativity" of random algorithms! They can do some wild shit! This is nothing new, we've been playing with these toys for like 70 fucking years. It's only recently that they started spewing words and everyone lost their minds over it.

[27183]

I used the example of 1G constant acceleration space flight in another thread which got downvoted to oblivion, but I think it's a good one. That's a technology we know how to build. We just need superconducting electronics and miniaturized fusion reactors, or a ship which is built like Project Orion to use nuclear bombs for propulsion.

Now write down a blueprint for superintelligence.

So I've given you two impossible engineering challenges, but one of them is feasible in principle because we at least have the tools to begin to tackle the theoretical calculations and therefore we can do engineering. We cannot do engineering on the superintelligence problem yet.

In my view it would be insane to believe we can build something that we can't even reliably imagine yet.

[derektank]

As early as the late 19th century, Louis Pasteur’s work had inspired a belief in the scientific community that it must, in principle be possible to selectively exterminate bacteria. The German physician Paul Ehrlich expounded on this in greatest detail in 1907 when he described his “magic bullet” (or Zauberkugel) theory for effectively targeting pathogens without harming the human host, similar to the immune system.

However, if you had had demanded someone for a blueprint in 1925 of how to design such a magic bullet, especially a magic bullet that targeted virtually all forms of bacteria, it would have sounded ludicrous. Yet, 20 years later, the world was manufacturing 6-7 trillion units of penicillin a year, capable of treating 3-6 million people. And that’s in spite of the fact that Fleming’s work sat mostly untouched for a decade before Howard Florey and Ernst Chain seriously set about to isolate and purify the substance.

You can quibble and say that penicillin was discovered, not designed, which is certainly true. But I would ask you to consider, does current AI development look more like design or discovery? Does it look more like analytical engineering or evolutionary selection? I would say on both counts the latter, in which case, we should prepare to be surprised how long it might take to make revolutionary advances. And that’s on both sides of the ledger, we might find ourselves stuck in the current paradigm for a long time. But, we might not be.

[27183]

Yes I think the drug discovery analogy is apt. I've spent a bunch of time playing with evolutionary algorithms, they're great fun. And when they work they can do surprising things! [edit] I think the drug discovery analogy does have some limits though. Drug discovery isn't a blind search through fitness space, it's informed by physics, chemistry, biology, and medicine. We have many guiding lights to illuminate the space and identify regions (still high-dimensional infinite regions!) that are likely to be productive. There are fewer lights to guide the way on a search for fitness in intelligence. Hell, we don't even know how to write down a decent objective function.

I wouldn't bet on evolving an intelligent, sentient being-in-a-box on a computer any time soon though. I'm of course prepared to be pleasantly surprised.

That said, I think it's pretty clear that LLMs are not going to get us there.

[davemp]

I don’t think people are arguing to stop researching AGI. Moreso against sales people trying to use the concept of AGI to sell products that are very much not AGI. Or devoting so many of our resources into such a pursuit that it causes harm to real people.

This is obviously complicated by the fact that LLMs/Agents are useful by themselves, but that’s not really the topic at hand.

[ThrowawayR2]

The parent poster argument boils down to "[something] is theoretically possible, therefore 1) it is guaranteed to practically implementable 2) in the reasonably near future". Both are simply prima facie false; one can ask an LLM to explain why if there's any doubt.

[techpression]

And now we’re desperately trying to ”upgrade” penicillin (and friends) because it doesn’t work any more in many cases. Do you think we can repeat the process or do we need something completely different?

This is why biological comparisons are weak, we talk about a few agents verifying and checking LLMs, meanwhile the world consists of almost an infinite number of the same, just operating on different time scales. I agree that with we don’t know the timescale, and we definitely don’t know if long term it will continue to work ”adding more of the same”. Throwing more penicillin at the problem sure as hell didn’t, but it looked great initially. And I’m obviously not arguing the human benefits of penicillin, just that what we thought would work forever quickly didn’t.

[27183]

To quote the great Dr. Malcolm:

> Life, uh, finds a way.

[stalfie]

No one imagined LLMs in their current format, it was simply a result of discovering that scaling compute and tokens produced better and better results with the Transformer architecture. The inventors of the Transformer architecture were working on better translation, and probably did not imagine that their architecture would lead to modern LLMs.

Imagining something in advance is not necessary at all for scientific advancement. This is particularily true in AI, and no one expects to imagine what superintelligence is until after it is created. You set up your datasets, your architecture tweaks, and measure the results on some set of benchmarks. There never was a blueprint, no plan beyond the experiment itself. We're not even close to understanding the things we have already created, and yet we created them. So why expect anything else for the next step?

[iugtmkbdfil834]

<< No one imagined LLMs in their current format

That is simply not accurate. There are examples of scifi novels, novellas and other media that dealt with it. We can argue over whether it was that exact format, implementation and so on, but that 'shape' ( to use a common llm term ) of technological advances was very much explored.

[stalfie]

I was referring more to the fact that no one predicted that next token predicting Transformers would go so far. Not about "AI" in general.

[27183]

> Imagining something in advance is not necessary at all for scientific advancement. This is particularily true in AI, and no one expects to imagine what superintelligence is until after it is created.

Then why does anyone expect to create it? I'll take a stab at an answer: they think an LLM is some kind of "incremental improvement" and therefore a step along the inevitable path to discovering AI. But that seems delusional to me. I can't imagine anyone sound of mind who knows how an LLM works thinks it's actually intelligent. So in what sense is it an "advancement" on the path to AI?

The concept of an incremental improvement in an objectiveless search in a high dimensional space is.. absurd.

[kalkin]

> actually intelligent

It's reasonable to doubt that LLMs are a path to AGI, but I don't understand how this is still a matter of dispute in 2026. What's your definition of intelligence that doesn't cover an entity that can translate fluently between dozens of languages and also solve open problems in mathematics? And be real-if you have one, is it a definition you or anyone would have given a decade ago, or are we doing "god of the gaps"?

[yusufozkan]

What is your definition of "actually intelligent"? I believe LLM's are more intelligent than the average human in a lot of ways according to the Legg/Hutter definition of intelligence: "Intelligence measures an agent's ability to achieve goals in a wide range of environments".

[stalfie]

No one knows how LLMs work. We know how the architecture works, but almost nothing about why. Saying "statistical next token prediction" tells you about as much about LLMs as saying "action potential thresholds" tells you about the brain. A true fact that explains very little.

And I'm sorry, but you're not up to date about interpretability literature, or for that matter philosophical discourse, if you think you have to be delusional to question whether LLMs are "intelligent", whatever you define that word to mean. The [latest publication](https://www.anthropic.com/research/global-workspace) from anthropics interpretability team purports that they see structures in Claude akin to those we think are associated with human consciousnesses experience in the brain. Are you going to dismiss the whole team as not being of "sound mind"?

Intelligence is a word with a somewhat unclear meaning to begin with, but you have move the goalposts pretty damn far to exclude LLMs at this point. They are certainly still lacking in some regards, but whether that disqualifies them for intelligence is very much a matter of debate.

[layla5alive]

I guess you don't really know how an LLM works then..?

[0xDEAFBEAD]

In this very thread I am being told that Fable is nothing but a bit of scale and refinement on well-known neural network techniques. And next I am told that we can't even imagine how to build superintelligence. Which is it folks?

[layla5alive]

Indeed, the real fun signal here is the conflict between how intelligent humans think they are and the different story told by much of human behavior, including within this very thread!

When an LLM makes a cognitive mistake, humans jeer "dumb stochastic parrot!" - when a human does the same, many are blind to it. Like a weird auto-Gell-Mann Amnesia - relative to reflecting on our own perceived cognitive "strengths" versus others weaknesses.

[victorbjorklund]

It’s also theoretically possible to travel almost at the speed of light. Doesn’t mean it’s rational to talk about it today as if imminent.

[lins1909]

How can not believing something that hasn't been proven be unscientific? Do you know that words mean things?

[ToValueFunfetti]

Science means the pursuit of knowledge. It doesn't mean "only believing proven things". If we're going to be rude, lets at least take the time to be right.

[trebaud]

You dont seem to understand what proof means. Human brain is made of matter, matter can be arranged to make a thing that reproduces human brain properties. What's the confusion here? I say its unscientific because it places the human brain beyond the scope of what can be operated on. Not having the knowledge or tech yet to achieve that is irrelevant since we have an existence proof.

[TheOtherHobbes]

Even if we accept your premise, and not everyone does, the confusion is whether we're capable of creating an equivalent arrangement, even in principle, using alternative materials.

That's not "irrelevant", it's fundamental.

[JumpCrisscross]

> Not believing that AGI is possible

One can simultaneously believe AGI is possible, be only modestly sceptical that our current methods are likely to yield it in the near term and still find the religious ferocity enveloping its discussion silly.

> saying that the human brain cognitive capabilities cannot be reproduced on other types of substrates is stupid at this point

Straw man. Nobody argued this. The discussion is around how urgent it is to policy treat a future hypothetical.

[ToValueFunfetti]

Nobody argued anything. GP just dismissed it as religion without engaging with a word of the material. Parent is taking a stab at why.

Not that I'm complaining. Cynicism is the failure mode I rely on HN for. It's the populism that's been getting to me.

[JumpCrisscross]

> GP just dismissed it as religion without engaging with a word of the material

Fair enough. I didn’t see anything novel in the article. So treating it as a motif within the abovequoted “Superintelligence: The Idea That Eats Smart People” context is fair and a real argument.

> Cynicism

Cynicism isn’t the opposite of blind optimism. Nihilism is. I’m not seeing a rejection of the article as being baseless as cynical or nihilist. It’s just pointing out a cultural thread that doesn’t seem to be useful.

[ToValueFunfetti]

I hate to get bogged down in semantics, but with the hope that one of the stronger top-level critiques makes it into the top slot here and this conversation gets buried:

Cynicism is defined as

>An attitude of scornful or jaded negativity, especially a general distrust of the integrity or professed motives of others.

I'm not saying that cynicism is automatically wrong, just that I once could trust that, when HN is wrong, it is due to cynicism applied in excess.

[oliculipolicula]

>..Nothing in life ... has any chance of succeeding without strenuous practice; and this is capable of overcoming anything..

---Diogenes (L) on Diogenes, arch-Cynic, VI.2.71

Afaik not formalised in Lean4, but sounds like optimism

Most of D's other stuff imho is optimism taken to its clear-eyed extreme. ya know. the way a very best version of us could always unironically RELY on HN [,after much practice]

>so those whose training has been of the opposite kind derive more pleasure from despising pleasure than from the pleasures themselves.

Might be why professional Nihilists admire Cynics (ie, for their ... unaffected comedy)

https://penelope.uchicago.edu/Thayer/E/Roman/Texts/Diogenes_...

[threetonesun]

I put it in the same bucket as living on Mars. Can it be done? Probably. Are we close? Not as close as people seem to want to believe. Is it a goal that will largely benefit society in its current form? Absolutely not.

[JumpCrisscross]

Eh, with Martian habitation we know what the roadmap looks like. With AGI we don’t. It could be proximate. Or it might not be. When it arrives, it could be totally economically uncompetitive outside the rich world. Or it could replace all human labour. Or progress to become a superintelligence.

We don’t know. Which makes proposing rules around it based on fiction more than science silly.

[neutronicus]

In particular, I just don’t buy into the “left behind unless” framework.

Perhaps Anthropic will create God in the Machine. Not foreclosing on that. But will it matter so much who was fucking around with Opus five e-folding times ago?

Either ClauDeus is benevolent and lifts you up (not left behind) or it isn’t, or not to you, and you are culled by a drone (left behind regardless).

Serenity Prayer time.

[gilrain]

> Not believing that AGI is possible is irrational and unscientific imo

This is an objectively wrong opinion.

[bwanab]

The other thing about this article is that it contains falsifiable claims that are, well, false. Consider the following direct quote: "The 2028 election cycle is heated, as usual. AI is the biggest topic." No poll of the American public has AI as one of the top concerns except in polls where the question is specifically about AI. The general polls show topics like healthcare costs, inflation/affordability, government dysfunction, and immigration as the top concerns. AI doesn't even show up in the top 20 in every list I looked at (Gallup, Pew, YouGov).

It sounds more like "everybody I know thinks AI is the top concern" when everybody they know is people like them, i.e. an echo chamber. It gives me no confidence in any other claims.

[Aurornis]

The AI 2040 writing and the AI 2027 before it both sprang from within the rationalist communities. Scott Alexander (of the popular rationalist Slate Star Codex / Astral Codex Ten blog) played a role in both articles, though he says he chose to leave his name off of this one.

AGI and AI doom scenarios have been evergreen topics in rationalism for years. Long before we had enough compute power to do anything at all useful with generative AI. Many of the rationalist scenarios about future AIs had very religious themes where the AI became so powerful that it was effectively a god.

Rationalists had concept's like Roko's Basilisk, a thought experiment where a future all-powerful AI might choose to punish anyone who predicted the arrival of AI but failed to contribute to the arrival of AI.

The rationalist communities took this thought experiment so seriously that the biggest rationalist forum of the time formally banned discussion of Roko's Basilisk for 5 years. They believed it was an "info hazard" because once you knew about this risk, you either had to contribute to bringing about this future AI god or you risked being punished by it in the future.

If this is new to you, you might think I'm exaggerating or making things up to make a group look bad, but this was such a core belief that it has its own Wikipedia page: https://en.wikipedia.org/wiki/Roko%27s_basilisk

The people writing AI 2027 and AI 2040 were immersed in these cultures. That's why they think that AI is the primary topic that anyone is talking about in elections, just like the Bitcoin people all thought crypto was the only relevant election topic.

The GP comment is getting attacked, but there really is a heavy religious fervor background to the AI fears in the rationalist community. The religious angle and extremism of the forums is downplayed when in crosses over into mainstream topics, but it's been there from the start.

[pnt12]

First I was intrigues by the rationalist movement, but I was deeply disappointed. To put it bluntly, seems to me like geeks LARPing as philosophers, and not with rational argument as they promise, but rationalizing their sci fi fanfictions.

[layla5alive]

Yes, calling themselves "rationalists" seems to be the biggest tell. That they wish they were that. Yet their behavior reflects anything but.

[ThrowawayR2]

If I recall correctly, the text of Yudkowsky's response to Roko's Basilisk used to be part of the Wikipedia page but was removed and not even a reference link was provided even though the response was mentioned in the body of the article. For posterity's sake, from https://rationalwiki.org/wiki/Roko%27s_basilisk/Original_pos... here it is in all its hilariously unhinged glory:

"One might think that the possibility of CEV punishing people couldn't possibly be taken seriously enough by anyone to actually motivate them. But in fact one person at SIAI was severely worried by this, to the point of having terrible nightmares, though ve wishes to remain anonymous. I don't usually talk like this, but I'm going to make an exception for this case.

Listen to me very closely, you idiot.

YOU DO NOT THINK IN SUFFICIENT DETAIL ABOUT SUPERINTELLIGENCES CONSIDERING WHETHER OR NOT TO BLACKMAIL YOU. THAT IS THE ONLY POSSIBLE THING WHICH GIVES THEM A MOTIVE TO FOLLOW THROUGH ON THE BLACKMAIL.

There's an obvious equilibrium to this problem where you engage in all positive acausal trades and ignore all attempts at acausal blackmail. Until we have a better worked-out version of TDT and we can prove that formally, it should just be OBVIOUS that you DO NOT THINK ABOUT DISTANT BLACKMAILERS in SUFFICIENT DETAIL that they have a motive to ACTUALLY BLACKMAIL YOU.

If there is any part of this acausal trade that is positive-sum and actually worth doing, that is exactly the sort of thing you leave up to an FAI. We probably also have the FAI take actions that cancel out the impact of anyone motivated by true rather than imagined blackmail, so as to obliterate the motive of any superintelligences to engage in blackmail.

Meanwhile I'm banning this post so that it doesn't (a) give people horrible nightmares and (b) give distant superintelligences a motive to follow through on blackmail against people dumb enough to think about them in sufficient detail, though, thankfully, I doubt anyone dumb enough to do this knows the sufficient detail. (I'm not sure I know the sufficient detail.)

You have to be really clever to come up with a genuinely dangerous thought. I am disheartened that people can be clever enough to do that and not clever enough to do the obvious thing and KEEP THEIR IDIOT MOUTHS SHUT about it, because it is much more important to sound intelligent when talking to your friends. This post was STUPID.

(For those who have no idea why I'm using capital letters for something that just sounds like a random crazy idea, and worry that it means I'm as crazy as Roko, the gist of it was that he just did something that potentially gives superintelligences an increased motive to do extremely evil things in an attempt to blackmail us. It is the sort of thing you want to be EXTREMELY CONSERVATIVE about NOT DOING.)"

[MaybiusStrip]

The current year is 2026. I think it's quite likely AI will make the top 3 by then. Putting this on my calendar.

[bwanab]

Yes, I can't predict the future, but if it were a bet on Kalshi I'd be willing to put my money on the AI won't make the list side.

[ToValueFunfetti]

This seems out of line with what you were saying above. Do you mean you would put money on a position less favorable than 50/50? What do you consider the likelihood such that you think you can announce it to be false today? 99%? I'd happily offer 30 to 1 odds on this today and add a 3x buffer for you up to $1000 of my dollars if you're serious.

[bwanab]

I'm confused. In the original comment I pointed out that the article claims AI is a major topic when polls show that isn't the case. I still believe that in 2028 AI will not be a major topic as shown by polls and voting behavior.

I'm not sure how I'd formulate the bet, but maybe it could be "as of election day 2028, AI will not be a top five concern mentioned in the most recent open-ended poll by Gallup polling the American public. This bet will not be decided by a poll in which AI is specifically mentioned in the polling question."

[ToValueFunfetti]

Not clear on where you're confused. The article does not claim this is the state today, it says in 2028- did you misread? In any event I think this is entirely plausible, say >9%, based on trendlines and the fact that I hear about this constantly already- two of the candidates in my primary had a position on this and I'm not exactly in SF. I'm prepared to hand you as much as $1k in the 91% of worlds where this doesn't happen in exchange for 30x as much (eg. 30k) in the 9% of worlds where it does. EV for me is .91 * -1 + .09 * 30 = +1.79. You claim you can say already today that this is false; I assume that means you're >99% certain. EV for you is then .99 * 1 + .01 * -30 = +0.69 (nice). Thus there is a mutually beneficial trade at the >3x your position and <1/3x my position point, and I offer 30:1 odds. I don't care if this is a nominal one of my dollars to thirty of yours, $1k is just my limit.

I don't normally do this, but you brought up your willingness to bet so I figured I'd offer.

I didn't think it was possible you misread- you had like 5 replies calling this out, but I recall them being gray and maybe you have showdead disabled- but don't feel obliged to commit if your real probability is 97% for this.

This term:

>This bet will not be decided by a poll in which AI is specifically mentioned in the polling question.

I cannot honor. In the >9% of worlds where you are wrong, any poll that excludes AI is necessarily heavily biased against me. Better to pick a particular poll.

[offnominal]

> falsifiable claims that are, well, false

Oh, I didn't realize it was 2028 already!

[somebodythere]

This prediction can't be scored until the 2028 election cycle.

You may think it's very unlikely the prediction will have turned out to be correct by the 2028 election cycle, but that is not the same thing as the prediction being scorable as false today.

[reasonableklout]

AI is not yet a top 5 concern for the American public, but polls show that it is for DC staffers: https://x.com/TimSchnabel/status/2072725548405518574

[fwipsy]

I remember thinking exactly the same thing around 5-7 years ago, in the GPT-2/GPT-3 era. "Oh sure they can produce semi-coherent output, but truly intelligent behavior is still far away. This isn't science fiction, they're just falling prey to Pascal's Mugging same as my religious friends did." Now I'm not so sure. I give the AI safety subculture as a whole a lot of credit for putting it on my radar back when it was otherwise still science fiction. I don't know if they're right about what comes next, but I think their case deserves to be evaluated on its merits, rather than assumed to be the result of psychological flaws.

> Yall, with your incredible wealth and resources you could do real good in this world and make society better, healthier, better educated, and the whole world more equal, just, and reduce the desperation and suffering. Reject the false and self-serving narratives that empathy doesn't matter, that altruism isn't "effective". You can change a person's whole life in a moment.

Confused at who this is directed towards. I'm fairly certain that the article was written by people who (at some point) identified as effective altruists, most of whom would enthusiastically agree with this. This community didn't start as AI researchers and later choose effective altruism; they were effective altruists who chose AI safety research as the most effective way to improve the world. Given that you apparently share their goals (a better world,) isn't it worth at least hearing them out on their methods?

[romanhounds]

>Given that you apparently share their goals (a better world,) isn't it worth at least hearing them out on their methods?

Their methods are about convincing others that things that enrich and empower themselves at the expense of others is "improving the world". This isn't the stance of serious people who want to improve the world.

[fwipsy]

Please don't confuse AI researchers and AI safety researchers (although there is some overlap.) That's like confusing the people from the Manhattan project with the people who protested 3 Mile Island, because they're both focused on nuclear technology. One effect of EA is that every AI researcher says they're trying to make the world better. Some of them are full of shit, but not all.

I'm fairly certain the authors would be happy to see AI shut down indefinitely. They just don't believe that the coordination problem is solvable. This is their best attempt to come up with something workable in the real world, or at least get people started thinking about it.

[nradov]

Right, the so-called "AI safety researchers" are worthless clowns and grifters with nothing of value to contribute. We would all be better off if they spent their time on something productive like writing entertaining sci-fi stories. They certainly seem to have the rich imagination necessary for that occupation.

[fwipsy]

Now this is an unserious comment, just throwing insults without explaining your reasoning at all. Do you believe that AI itself is worthless? Or that it's just incapable of causing harm? Do you resent the safeguards placed on AI because they are personally inconvenient to you? When introducing a new technology at this scale, don't you think it's worth looking before leaping?

[nradov]

So called "AI safety researchers" are themselves unserious so of course I don't take them seriously. LLMs are extremely useful, I use them constantly for a wide variety of tasks. LLMs don't cause harm: humans cause harm. There's no real need to intentionally cripple or restrict the models in the name of "safety"; that's just stupid and done only to prevent corporate embarrassment.

[nadir_ishiguro]

Sadly they're very bad writers. Have you actually read the stuff they produced? Horrible. Yudkosky himself wrote https://en.wikipedia.org/wiki/Harry_Potter_and_the_Methods_o...

It's so bad

[reasonableklout]

I'm confused, the authors of the website are arguing that the development of AI should be slowed down so that society can adapt. What do you think they are trying to convince others of?

[MaybiusStrip]

Which one of the authors of the website are you referring to?

[heyts]

> they were effective altruists who chose AI safety research as the most effective way to improve the world

This actually made me laugh, I’m sorry

[grey-area]

Effective altruism is also very attractive to manipulative sociopaths who want to maximise their power over others whilst appearing virtuous and hoarding wealth and power. Poster boy for this movement is the convicted fraudster SBF. I believe Altman is also a fan.

As to a better world or super intelligence, I’ll believe it may be possible when I see some signs of intelligence from what people are calling AI, instead of plausible text and image generation based on a very large corpus.

[0xDEAFBEAD]

>I believe Altman is also a fan.

Altman says he thinks it is an "incredibly flawed movement"

https://x.com/sama/status/1593046526284410880

The dislike is mutual. Here's a long video takedown of Sam from a major EA org:

https://www.youtube.com/watch?v=_eYTkvZqbnQ

[HDThoreaun]

And what movement isn’t attractive to manipulative sociopaths who want to maximize their power? That’s unavoidable when dealing with humans.

[notahacker]

An important adjacent point is the same people that are insisting that only their research can stop intelligent manipulative computers from controlling the human race are also some of the few people who believed that OpenAI was a philanthropic endeavour and that Sam Bankman Fried was trustworthy...

[fwipsy]

I'm not seeing any claims to exclusivity. I'm seeing people actively encouraging more people to enter the field of AI safety/security and openness to robust debate, etc.

I agree that there seems to be a huge problem with value drift--either people who used to research AI safety, pivoting to building AI (looking at Anthropic) or people who only ever paid lip service to safety (I tend to put Musk and Altman in this category.) These people need to be held accountable, but it doesn't mean every AI safety researcher ever was a stooge or a fraud.

[notahacker]

Ok, fair enough theyre not entirely opposed to outsiders setting up research institutes like theirs and enjoy debate (or the optics of debate) more than most, and I don't think most of them are insincere. But the point wasn't that they were exclusionary, but more that whilst priding themselves on their supposedly superior ability to predict the future and "align" AIs, they somehow missed the value drift and human non-alignment with their values basically any outsider saw coming...

[HDThoreaun]

OpenAI now controls one of if not the largest philanthropic endowments on the planet. They are still a philanthropic endeavor, though I agree the non profit bait and switch as well as the Altman board situation were tragedies.

[notahacker]

I mean, they're also gearing up for one of the largest IPOs in history to generate unprecedented amounts of wealth for themselves, and the nonprofit foundation's focus is community engagement with OpenAI products. Goldman Sachs has a large philanthropic endowment but I wouldn't confuse that with their priorities being philanthropy.

The point was I think pretty much everyone else saw the bait and switch coming...

[grey-area]

Well I think there is a definite pattern to sociopaths using a movement that seems externally good and has many true believers - like say an open foundation to promote AI, or effective altruism. Then twisting it to their own ends to gain power, influence and wealth.

We should therefore be very skeptical of people claiming altruistic motives while requiring that they be in control.

[danbruc]

I remember thinking exactly the same thing around 5-7 years ago, in the GPT-2/GPT-3 era.

ChatGPT was announced three and a half years ago, 30-11-2022.

[semiquaver]

Do you think that ChatGPT is when people on hacker news first became aware of the GPT series of models?

https://hn.algolia.com/?dateEnd=1594339200&dateRange=custom&...

[IshKebab]

And GPT-3 was released 6 years ago in 2020... What's your point?

[MichaelDickens]

I would rather see you engage with the substance of the article rather than skipping right to insulting the authors. I don't think this sort of comment is up to the standard I have come to expect from HN.

[romanhounds]

> I intend to donate (at least) 20% of my lifetime income to effective charities. I publish my donations on my Donations page.

From your bio I suspect you're already in the cult.

[johnfn]

When your best counter-argument is “you donate money to charity” you have to start wondering if you’re on the bad side and not the good side.

[0xDEAFBEAD]

"I see your objection to ad hominem arguments, and raise you a further ad hominem argument"

[nadir_ishiguro]

His whole blog is exactly the same kind of nonsense as the article and reeks of effective altruism.

That's not an ad hominem.

[0xDEAFBEAD]

Attacking a person instead of the substance of what they're saying is ad hominem.

[HDThoreaun]

Imagine thinking someone donating money is evidence of something bad

[nadir_ishiguro]

If the cause they donate to is bad, it is

[jamilton]

What is bad about "effective charities"? It's probably mostly for malaria nets!

[kalkin]

> Reject the false and self-serving narratives that empathy doesn't matter, that altruism isn't "effective"

Who is this supposed to be arguing with? It sort of reads like it's trying to disparage "effective altruism", but I'm not sure.

Setting aside any of the AI stuff, I've started to find it pretty grating when people seem to imply that transferring millions of dollars from wealthy people in California and the UK to impoverished Kenyans and Rwandans, or buying malaria bednets which can save a child's life for the cost of a fancy new gaming rig, is "self-serving" or something because weirdos are doing it, while true caring for other people involves [unspecified thing that doesn't appear to ask any material sacrifice comparable to donating a large percentage of income].

[huhkerrf]

> I grew up in evangelical christianity, and to them the end of the world is just around the corner

I grew up in evangelical Christianity, I'm still an evangelical Christian, and you're wrong to paint with a broad brush like that. Are there evangelicals that are millenarian? Yep, of course.

But that is not my experience across multiple evangelical churches across decades.

[bryan0]

From TFA

> Plan A is primarily a recommendation, not a prediction.

That sounds nothing like “religious fervor”

The fact that technology can increase existential risk for civilization is not fantasy. It’s a risk that should be reasonably discussed.

[fulafel]

To interpret charitably, I guess we could solve this for religion = technological development so advanced as to be indistinguishable from magic. It's been done in some Star Trek episodes I think.

[taurath]

It has a lot of the properties of magic. You can put on quite a magic show with a trillion dollars.

[0xDEAFBEAD]

Imagine observing powered flight for the first time and saying: "This is religious fervor folks. I remember the mythological story of Icarus from when I was younger. Key word, mythological."

The existence of mythology describing Scenario X is not a valid argument against the plausibility of Scenario X.

If we can acknowledge the possibility of nuclear doomsday without running an RCT of sample size 100 Earths, 50 of which undergo nuclear Armageddon, to verify that nuclear holocaust indeed a real phenomenon... then we can do the same for AI. Understand the arguments being made instead of engaging in these guilt-by-association arguments.

Modern AI capabilities are already mind-boggling by the standards of 20 years ago. We should at least prepare for the possibility that trends continue on the current trajectory.

[reactordev]

The end of the world is always just out of reach. It’s the perfect carrot on a stick for people who are conditioned to be afraid.

[derektank]

>Reject the false and self-serving narratives that empathy doesn't matter, that altruism isn't "effective". You can change a person's whole life in a moment.

You can change more people’s lives, more substantially, if you donate effectively. Effective altruism started out as (and the majority of effective altruist financing is committed to) an effort to rationalize what has historically been a very emotionally driven activity by deploying insights from developmental economics. If you want to take longtermists to task, go right ahead, but please refrain from torching anti-malarial or child vaccination programs while doing so.

[gcr]

But in the public image, the EA community is synonymous with doubling down on AI / AGI to the exclusion of the other projects.

OpenPhil changing its name to Coefficient Giving, 80000 hours and bluedot and (to a lesser extent) CFAR dropping other initiatives and switching to AGI promotion… to my knowledge GiveWell is the only other big name that continues to advance other initiatives. Then look at figureheads like SBF committing fraud and begging for a pardon from the architects of the USAID shutdown… We begin to paint a picture of a community that’s (by and large) abandoned its principles for power.

I know the view from the inside is more nuanced, but I think it’s a reasonable association for random members of the public to make.

My critique of the EA community is that it’s myopic and unregularized. If you really think AGI is make-or-break for civilization, it’s completely rational to deprioritize side bets.

[derektank]

>My critique of the EA community is that it’s myopic and unregularized. If you really think AGI is make-or-break for civilization, it’s completely rational to deprioritize side bets.

I’d be curious to hear you expand on this. What binds the EA community together, from the shrimp welfare enthusiasts and wild animal initiative, to the longtermist lightcone obsessive, to the people funding vitamin A supplementation, is simply a commitment to maximizing the number of quality adjusted life years saved each year and a belief that empirical observation can be used to improve that number.

To my mind, this is a valuable insight on its own. Yes, if you come to such a heuristic with absurd prior beliefs, such as whether 100k neurons alone have QALYs in the first place or by placing equal value on people actually alive today and hypothetical people in the far flung future, you will get absurd results. Garbage in, garbage out. But that’s not an indictment of the fundamental insight, especially when you consider how poorly allocated the roughly $2 trillion in global charitable spending is.

[mekael]

As a society we already know how to make the lives of everyone better:

1. Stable housing

2. Access to safe drinking water

3. Access to food

4. Access to healthcare

5. Access to education

6. Stable governments

The EA community has so many "ideas" about what would help, when all they need to do is focus on those six and the world would be as close to a utopia as you or I could hope to see in our lifetimes.

I legitimately thought you were kidding about the shrimp welfare initiative, but after looking it up I was more infuriated about EA than I normally am when it comes up. I can think of several causes which would be better served with 3 million USD, and all of them take care of human beings. Living, breathing, intelligent human beings. These people should not be in any position of power or taken seriously ever.

[derektank]

There are tradeoffs to make between those six items. Money spent on stable housing is money not spent on safe drinking water is money not spent on education. There is a limited amount of money spent on charitable giving every year, and we obviously cannot afford to provide all of those things to everyone with the funds currently available. If you’re arguing that people should simply be donating more money, I (and I think every EA) is right there with you. But given the world exists as it is, I would rather see money that’s currently being donated diverted from Catholic missionary schools and good governance NGOs to GiveWell’s top charities because I’m very confident that it would mean fewer dead kids.

[gcr]

I agree with your last sentence, but we can’t partition the budget up so cleanly though. Societal spending is interdependent and additive across domains. Spending on cheap access to safe water or food reduces healthcare costs. Spending heavily on education tends to strengthen government in the long run, and better-run governments amplify efforts to improve housing/food/water/.

The one exception is AI. Historically, redirecting charitable funds towards AI safety tends to starve or undo the rest of these efforts, which is why I’m so disappointed by many EA institutions dropping other initiatives to put their eggs in the AGI basket.

[ls612]

AI 2027 made substantive predictions about the near term future of the technology and the implications of it. I think the worst you can say about it is that the Agent-1 moment might come next year rather than this year. But being off by a year is far better than this 2040 slop, which is mostly disconnected from reality.

The one thing I will say that they are correct about is that AI does have the potential to be highly destabilizing geopolitically, even if they get everything downstream of that wrong.

[pibaker]

When I learned that this website is by the same people who gave us AI 2027, I immediately thought about the Wikipedia page on doomsday predictions.

https://en.wikipedia.org/wiki/List_of_dates_predicted_for_ap...

Notice the many times when a prediction failed and the so called prophet would come back in a few years and give you a new date. And they would tell you, well, this time it's real.

I do find it ironic that many of the AI predictions are coming from the self titled "rationalists." It seems like building your identity around being rational and immune to psychological pitfalls is a good way to ensure that you don't even notice that you have walked straight into the one psychological trap every cult has employed since time immemorial.

[ctoth]

> When I learned that this website is by the same people who gave us AI 2027, I immediately thought about the Wikipedia page on doomsday predictions.

Then, you opened the page and read it and realized that this prediction was contingent? You know what a conditional is (I would assume) if this, than that?

Then you realized that the only reason you were posting this comment was as a sort of silly gotcha "Oh look at the guys who keep increasing the number" instead of talking about the differences between the scenarios?

Then what happened?

[thegrim33]

They gave a ~2 year timeframe for their predictions in this one (tied to the next presidential election), so in 2 years when none of this has happened will they then switch over to a new AI 2044 Plan? Where's the accountability? Where's the followup or retrospective? What if there was some mechanism that branded these people so that when they made claims in the future people could clearly see the authors labeled as "made completely wrong fantastical claims in the past"?

[0xDEAFBEAD]

"Doomsday predictions have occurred since time immemorial. Ergo, the Cuban Missile Crisis is nothing to worry about."

You actually have to look at the substance of the prediction. Sorry.

[defgeneric]

Like all forms of eschatology it's not actually concerned with the future to come--it's real aim is to have influence now.

It's just a form of rhetoric.

To dismiss them, saying, "ah ha! they're making the same error as all failed prophets!" misses the point.

What they are really talking about is AI governance.

[dheera]

... and then there are the actual planet-threatening astronomical events that humanity should think about ways to mitigate but I'm worried that human lifespans and capitalism prevent us from working towards mitigations. Everyone will just say "meh, 1 million years is a long time, I won't be around" and a million years will go by.

https://en.wikipedia.org/wiki/Timeline_of_the_far_future

[thegrim33]

Capitalism, in particular NASA's commercial space development program, have already provided and demonstrated initial asteroid redirect capability via the DART mission launched by SpaceX, which impacted an asteroid and measured the changes to its trajectory.

[ozozozd]

I completely agree. But not entirely sure that triggering the EA crowd is the effective way to deliver this message.

They are the crowd who need to understand your framing the most, but they’re completely shutdown by your framing, as any religious follower would be.

Your message gives me hope that not everyone’s drunk on the kool aid.