[trebaud]

Or you deeply suffer from normalcy bias. Not believing that AGI is possible is irrational and unscientific imo, the human brain exists, it is not made of magic, it can reproduced. It is as simple as that. We can debate about timelines, architectures that may lead us there, etc... You can even talk all day long about definitions of AGI, and people waste their time doing that. But saying that the human brain cognitive capabilities cannot be reproduced on other types of substrates is stupid at this point.

[davemp]

There is a vast gulf between theoretically possible and technologically feasible.

If you can’t provide a realistic path to achieve something, you’re asking people to believe in science fiction.

You could tell me that a rock’s molecules are comprised of protons, neutrons, and electrons. Blood is also entirely protons, neutrons, and electrons; so theoretically, one could rearrange stone into blood. But without an actual method to do so, it sounds like you’re telling me that you can squeeze blood from a stone.

> the human brain exists, it is not made of magic, it can reproduced

Yeah. It only takes 9 months and ~18 years of training…

> But saying that the human brain cognitive capabilities cannot be reproduced on other types of substrates is stupid at this point

Let’s be clear. Everyone is talking about silicon transistors here. That’s what we’ve got.

Digital computers have real limits. Sensors and other sources of training data have real limitations. It’s not clear that we can organize them in a way to reproduce organic brains.

[themgt]

What's strange to me about these comments is they're timeless. They could have been written in 2026 or 2016 or 1966.

Like, afaict, for many on HN going from ELIZA->Fable 5 just didn't cause any update to priors regarding this whole philosophical question. The argument against has remained unchanged. I don't see any point in arguing about it, I just find it very strange.

[defgeneric]

It's a form of denial. We're getting another "de-thronement of man" on the order of Copernicus and Darwin. Some get excited, others turn away in horror. Negation is the outward expression of the desire to keep human intelligence wrapped in its mystical veil.

One popular idea is that these systems will asymptotically approximate human intelligence because they're trained on mostly human-written texts. Not only is that untrue, it's also directly contradicted by our experience with previous RL-systems, where they seem to breeze right by human ability without even the slightest hiccup.

[disgruntledphd2]

> with previous RL-systems

Most human systems are much, much, much more complicated than most closed world games (which is where RL approaches have seen massive success, mostly through self-play).

Like LLMs are great, but I honestly can't see us getting actual general intelligence out of them.

[layla5alive]

They already possess general intelligence by many metrics. Sure they miss a few, but that's nitpicky and goalpost shifty - lots of humans make errors of all sorts as well, or incur brain damage limiting them in one or a few areas of intelligence - we don't then say they are not general intelligence anymore.

I think maybe you mean superintelligence, which is a more fair critique.

[disgruntledphd2]

> They already possess general intelligence by many metrics.

Can you share the metrics you are using for this assessment?

They are really powerful tools, but a quick glance at their thinking tokens (which is a bad name, tbh) rapidly disabuses me of the notion that they are general intelligences.

They possess large amounts of crystallised intelligence (i.e. they have a lot of knowledge), but their fluid intelligence is definitely lower than the human median.

[iugtmkbdfil834]

<< One popular idea is that these systems will asymptotically approximate human intelligence because they're trained on mostly human-written texts.

Can you elaborate on this? I am clearly not aware of this line of thinking and the related contradiction.

[defgeneric]

Approximate as a limit and not surpass, where the hidden hope actually seems to be that they don't surpass. There are lots of variations floating around, from simple metaphors like "LLM as librarian that speaks to you," to even Sutton's remark that rather than being a case where we've taken the "bitter lesson" to heart, LLMs may be a yet another case where we're limited by it.

[beepbooptheory]

Unpack this a little bit. Why is it strange or interesting to you? What specific priors need to be updated for us here? What is the philosophical questions at play for you?

[themgt]

To meta-unpack a little bit ... it is strange to me that Fable is far more capable of discussing these questions than apparently 99% of humans. Along with being more capable at quite a lot else than most humans.

[notahacker]

It doesn't seem at all strange to me that a chatbot trained by true believers in an AI singularity and the importance of safety guardrails will give more satisfying answers to true believers in an AI singularity and the importance of safety guardrails than talking to humans who might ask questions they're not prepared to answer (or might say nasty sceptical things or just not seem interested)

As for "updating priors", that goes both ways. There's plenty more reason to think "hey, transformers and RLHF might actually make some killer products" but certainly no reason to think the few people who didn't realise that "GPT3 is too dangerous to release" and "all software engineers will be replaced within 6-12 months" were marketing rather than prophecy have some kind of special insight into how it's all going to pan out. Clock's ticking to the promised 2027 reckoning too...

[themgt]

OK. Anyhow ... if there's a cognitive task you are personally superior to Fable at, let us know.

[beepbooptheory]

OK.. when you discuss these things with your Fable, what topics come up? Can you articulate one of the questions? I am probably just another dumb human FYI, but just try it out and we will see if I can follow along.

[zdragnar]

Fable 5 doesn't represent anything new, other than scale and some refinement techniques, over the original LLMs. In the chase for AGI specifically, LLMs are a dead end, just like all the other AI technologies that died in the AI winter.

What priors should be updated?

[0xDEAFBEAD]

>Fable 5 doesn't represent anything new, other than scale and some refinement techniques, over the original LLMs.

Yet it is generating billions in revenue which Eliza did not.

Perhaps all we need is scale and some refinement techniques to eat a big fraction of the economy.

If unimpressive inputs lead to impressive outputs, that should make you more worried, not less.

[zdragnar]

The commercial viability is orthogonal to whether it achieves AGI, which is effectively what "reproducing the human brain" amounts to in this discussion.

[azan_]

What's your definition of AGI?

[m4rtink]

And possibly trilions in running costs, not mentioning all the shady training data sources.

[dwaltrip]

Strong disagree. Fable is first model that actually feels smarter than me in certain non-trivial ways.

It can hold many complex and partially contradictory thoughts in its head at once, in a way that feels significantly superior to Opus (for example). And then can make reasonable syntheses across these.

In a couple rounds of back and forth, with relatively low effort (but strategic) prompting, it produces complex, accurate analyses in 5-10 minutes that would take me multiple hours of hard, very focused work.

I still need to remain tightly in the loop, providing frequent course correction, clarification, high level reframing, nudging, and grounding.

It incorporates my feedback incredibly well.

It’s honestly staggering. Fable has changed my assessment of the current trajectory more than any model since possibly gpt-4. Opus 4.5 of last year might be a close second.

———

My advice for anyone who wants to get more value out of these tools:

When a model does something idiotic, don’t throw your hands up in the air. Be curious. Try to turn it into a puzzle to be solved.

It know it’s hard sometimes, especially if you are drowning in slop from other people… or generated by yourself, heh.

It can be exhausting. I struggle with this also. I have thoughts on how to make it better. We shall see.

[layla5alive]

Human brain looks a lot like scale and minor refinement above dog brain. Or mouse brain.

[iugtmkbdfil834]

I just realized that you might really be onto something. I wonder now if it is just a function of our very human inability to let go of a known construct that has served us well until now or something else. I have my own opinions, but as strange as it sounds, this may be the HN equivalent of ok boomer moment.

[throw310822]

Jesus... This morning while I was drinking coffee and staring at the screen (it's Saturday) an agent did the equivalent of days of my work, reading code, understanding, hypothesizing, comparing, using tools, writing scripts, launching compilers and running tests, identifying problems and proposing solutions, and more. Only someone who hasn't spent a second reflecting about what it means to think and to be intelligent can claim that we miss a realistic path to intelligence. It's so damn clueless and stubborn and confidently wrong that it annoys me immensely, so sorry for the rant.

[davemp]

> clueless and stubborn and confidently wrong

Uhuh. I really shouldn’t be replying to this type of comment from a throwaway.

But the extremely powerful semantic search that we get from LLMs isn’t enough. I don’t think anyone is credibly arguing otherwise?

Agents already are a layer on top trying to bridge the gap. But they’re really just using LLMs as a heuristic to explore extremely NP problem spaces. The notable successes with agents so far are when we can provide them with a solid verifier and preferably additional context hints on the steps to take in the problem space. See the test oracle problem on where this gets us.

So forgive me if I think that it would be enough of a jump in computational complexity to remove those guard rails that it’s not feasible. But don’t say that I’m clueless, stubborn, or confidently wrong.

[27183]

> reading code, understanding, hypothesizing, comparing, ...identifying problems and proposing solutions, and more

Except it did none of those things, really, because that's not how it works. This might help, it's a good writeup: https://www.0xkato.xyz/how-llms-actually-work/

We know how these machines work, it's not mysterious, there's nothing "extra" happening.

[eihli]

Now show me a writeup that explains how the brain works so I can understand why the brain does those things.

> We know how these machines work, it's not mysterious, there's nothing "extra" happening.

It sounds like you're saying "We don't know how brains work, they're mysterious, there's something 'extra' happening", and using that as justification for why you're saying a computer, an AI, can't "understand".

I think most people on Hackernews now who would use the phrase "my AI worked overnight and hypothesized, compared, etc..." already know how an LLM works, and still chooses to use those words. So the issue isn't that they don't understand. It's that they understand and still use those words. So the disagreement is somewhere else.

[27183]

I'm not claiming there's something "extra" happening in brains. Merely that we just don't know how they work well enough to use that knowledge to do engineering. Neural nets are quite unlike brains, despite the unfortunate shared vocabulary.

OTOH we do know how neural nets work, and they definitely don't do "thinking" or "reasoning".

[fragmede]

First you have to give a specific definition for thinking and reasoning before determining if they definitely do or don't do such a thing.

[sudb]

This feels like a semantic disagreement to me? If an LLM got to an acceptable end result code-wise, what would you call the process that took place to get it there?

[27183]

I'd call it what it is: a good enough stochastic search result extracted from the model's embedding space.

[sudb]

Is an implication of this that models are incapable of producing entirely novel code?

Also, not to get too reductionist about this, but what do you posit is special about what is happening when humans think? Intelligence is hard to define so clearly, I reckon.

[0xDEAFBEAD]

Just a few comments ago you argued that we don't know how to build superintelligence. Now you're saying we know how the (unevenly superintelligent) Fable system works.

It doesn't seem like you're being consistent here. I'm concerned there might be some motivated cognition going on.

"What is true is already so. Owning up to it doesn't make it worse. Not being open about it doesn't make it go away. And because it's true, it is what is there to be interacted with. Anything untrue isn't there to be lived. People can stand what is true, for they are already enduring it."

[27183]

I'm not following the "Fable" stuff, what is that? [edit: ah it's a new language model from Anthropic.. yeah still don't see how it's relevant here]

[throw310822]

> We know how these machines work, it's not mysterious, there's nothing "extra" happening.

Lol. This is more telling about your implicit unscientific preconceptions that you wanted to reveal. Of course there isn't anything "extra". Where do you think intelligence comes from, some mysterious realm? It's physical, computational. The fact that at the bottom we produced it via matrix multiplication is irrelevant. Maybe humbling. You are denying a visible fact (a machine performs tasks that require flexible analytical and cognitive skills) precisely because there is no magic happening anywhere.

[27183]

> Where do you think intelligence comes from, some mysterious realm? It's physical, computational.

Well, no. I don't think it comes from some mysterious realm. I think that which is not physical does not exist [edit: and if you like I'll follow that one right down the rabbit hole--continuity and infinity are useful delusions]. But that eminently does not mean we know what intelligence is, let alone how to build one.

> The fact that at the bottom we produced it via matrix multiplication is irrelevant.

Huh? We don't even know what "it" is. How can you say you produced it?

> a machine performs tasks that require flexible analytical and cognitive skills

You see that, I see a lucky stochastic search result. Don't underestimate the "creativity" of random algorithms! They can do some wild shit! This is nothing new, we've been playing with these toys for like 70 fucking years. It's only recently that they started spewing words and everyone lost their minds over it.

[throw310822]

You do realise then that "we know how it works, there is no "extra" there" is an argument that can be used against any artificial intelligence, now or in a thousand years, as well as (at some level) against human intelligence (no magic, it's all physics, just dumb cells exchanging signals). This should be enough to give you pause- you immediately reached for an argument that is entirely empty.

> I see a lucky stochastic search result

Again you're reaching for a mechanistic explanation of some kind (let's leave for the moment whether it makes sense or not) as if having an explanation somehow contradicted a display of intelligence. It doesn't. Yes of course we made it, we know how it works (ar some level) and there is no magic. But what matters is the result- this machine, matrix multiplier, stochastic parrot, consistently displays intelligence, to the point of being able to perform very complex, open-ended tasks that integrate discovery, planning, tool usage, decision and even some aesthetic sense, understanding and using natural language, context awareness, you name it.

> This is nothing new, we've been playing with these toys for like 70 fucking years

Lol no. For god's sake. Hundreds of billions of parameters organised in a specific architecture and trained with unimaginable amounts of data and compute? Unless by "these toys" you mean "any computer program vaguely AI-related".

[27183]

I used the example of 1G constant acceleration space flight in another thread which got downvoted to oblivion, but I think it's a good one. That's a technology we know how to build. We just need superconducting electronics and miniaturized fusion reactors, or a ship which is built like Project Orion to use nuclear bombs for propulsion.

Now write down a blueprint for superintelligence.

So I've given you two impossible engineering challenges, but one of them is feasible in principle because we at least have the tools to begin to tackle the theoretical calculations and therefore we can do engineering. We cannot do engineering on the superintelligence problem yet.

In my view it would be insane to believe we can build something that we can't even reliably imagine yet.

[derektank]

As early as the late 19th century, Louis Pasteur’s work had inspired a belief in the scientific community that it must, in principle be possible to selectively exterminate bacteria. The German physician Paul Ehrlich expounded on this in greatest detail in 1907 when he described his “magic bullet” (or Zauberkugel) theory for effectively targeting pathogens without harming the human host, similar to the immune system.

However, if you had had demanded someone for a blueprint in 1925 of how to design such a magic bullet, especially a magic bullet that targeted virtually all forms of bacteria, it would have sounded ludicrous. Yet, 20 years later, the world was manufacturing 6-7 trillion units of penicillin a year, capable of treating 3-6 million people. And that’s in spite of the fact that Fleming’s work sat mostly untouched for a decade before Howard Florey and Ernst Chain seriously set about to isolate and purify the substance.

You can quibble and say that penicillin was discovered, not designed, which is certainly true. But I would ask you to consider, does current AI development look more like design or discovery? Does it look more like analytical engineering or evolutionary selection? I would say on both counts the latter, in which case, we should prepare to be surprised how long it might take to make revolutionary advances. And that’s on both sides of the ledger, we might find ourselves stuck in the current paradigm for a long time. But, we might not be.

[27183]

Yes I think the drug discovery analogy is apt. I've spent a bunch of time playing with evolutionary algorithms, they're great fun. And when they work they can do surprising things! [edit] I think the drug discovery analogy does have some limits though. Drug discovery isn't a blind search through fitness space, it's informed by physics, chemistry, biology, and medicine. We have many guiding lights to illuminate the space and identify regions (still high-dimensional infinite regions!) that are likely to be productive. There are fewer lights to guide the way on a search for fitness in intelligence. Hell, we don't even know how to write down a decent objective function.

I wouldn't bet on evolving an intelligent, sentient being-in-a-box on a computer any time soon though. I'm of course prepared to be pleasantly surprised.

That said, I think it's pretty clear that LLMs are not going to get us there.

[davemp]

I don’t think people are arguing to stop researching AGI. Moreso against sales people trying to use the concept of AGI to sell products that are very much not AGI. Or devoting so many of our resources into such a pursuit that it causes harm to real people.

This is obviously complicated by the fact that LLMs/Agents are useful by themselves, but that’s not really the topic at hand.

[ThrowawayR2]

The parent poster argument boils down to "[something] is theoretically possible, therefore 1) it is guaranteed to practically implementable 2) in the reasonably near future". Both are simply prima facie false; one can ask an LLM to explain why if there's any doubt.

[techpression]

And now we’re desperately trying to ”upgrade” penicillin (and friends) because it doesn’t work any more in many cases. Do you think we can repeat the process or do we need something completely different?

This is why biological comparisons are weak, we talk about a few agents verifying and checking LLMs, meanwhile the world consists of almost an infinite number of the same, just operating on different time scales. I agree that with we don’t know the timescale, and we definitely don’t know if long term it will continue to work ”adding more of the same”. Throwing more penicillin at the problem sure as hell didn’t, but it looked great initially. And I’m obviously not arguing the human benefits of penicillin, just that what we thought would work forever quickly didn’t.

[27183]

To quote the great Dr. Malcolm:

> Life, uh, finds a way.

[stalfie]

No one imagined LLMs in their current format, it was simply a result of discovering that scaling compute and tokens produced better and better results with the Transformer architecture. The inventors of the Transformer architecture were working on better translation, and probably did not imagine that their architecture would lead to modern LLMs.

Imagining something in advance is not necessary at all for scientific advancement. This is particularily true in AI, and no one expects to imagine what superintelligence is until after it is created. You set up your datasets, your architecture tweaks, and measure the results on some set of benchmarks. There never was a blueprint, no plan beyond the experiment itself. We're not even close to understanding the things we have already created, and yet we created them. So why expect anything else for the next step?

[iugtmkbdfil834]

<< No one imagined LLMs in their current format

That is simply not accurate. There are examples of scifi novels, novellas and other media that dealt with it. We can argue over whether it was that exact format, implementation and so on, but that 'shape' ( to use a common llm term ) of technological advances was very much explored.

[stalfie]

I was referring more to the fact that no one predicted that next token predicting Transformers would go so far. Not about "AI" in general.

[27183]

> Imagining something in advance is not necessary at all for scientific advancement. This is particularily true in AI, and no one expects to imagine what superintelligence is until after it is created.

Then why does anyone expect to create it? I'll take a stab at an answer: they think an LLM is some kind of "incremental improvement" and therefore a step along the inevitable path to discovering AI. But that seems delusional to me. I can't imagine anyone sound of mind who knows how an LLM works thinks it's actually intelligent. So in what sense is it an "advancement" on the path to AI?

The concept of an incremental improvement in an objectiveless search in a high dimensional space is.. absurd.

[kalkin]

> actually intelligent

It's reasonable to doubt that LLMs are a path to AGI, but I don't understand how this is still a matter of dispute in 2026. What's your definition of intelligence that doesn't cover an entity that can translate fluently between dozens of languages and also solve open problems in mathematics? And be real-if you have one, is it a definition you or anyone would have given a decade ago, or are we doing "god of the gaps"?

[27183]

I can't give you or your sibling a better answer than "you'll know it when you see it". Some people see it now. I think they're wrong, because it seems like the results you're describing are easily explained by fuzzy search in the space of embeddings and then forming strings of plausible tokens related to the resulting region of embeddings space. In other words, the things we know LLMs actually do.

That's more or less looking for interesting patterns in a jpeg or another lossy compression result. It's interesting that the models seem to be able to (fairly) reliably return relevant chunks of the image. Even more interestingly, they seem to be able to invent plausible chunks of image that aren't even there. That doesn't meet my bar for intelligence though. I'd need to see it learn and adapt. I'd need to see it be clever, not merely "knowledgeable". I'd need to see it capably analyze itself. I'd need to see it reasonably estimate uncertainty and know itself in the sense that it has some idea how right or wrong it is about something. I'd need to see it exercise judgment.

I don't think I'd give a different answer a decade ago but who knows.

[edit] For all we know, one of the salient features of intelligence is that intelligent beings are incapable of precisely defining it. I'm not sure how productive it is to attempt to do so.

[yusufozkan]

What is your definition of "actually intelligent"? I believe LLM's are more intelligent than the average human in a lot of ways according to the Legg/Hutter definition of intelligence: "Intelligence measures an agent's ability to achieve goals in a wide range of environments".

[stalfie]

No one knows how LLMs work. We know how the architecture works, but almost nothing about why. Saying "statistical next token prediction" tells you about as much about LLMs as saying "action potential thresholds" tells you about the brain. A true fact that explains very little.

And I'm sorry, but you're not up to date about interpretability literature, or for that matter philosophical discourse, if you think you have to be delusional to question whether LLMs are "intelligent", whatever you define that word to mean. The [latest publication](https://www.anthropic.com/research/global-workspace) from anthropics interpretability team purports that they see structures in Claude akin to those we think are associated with human consciousnesses experience in the brain. Are you going to dismiss the whole team as not being of "sound mind"?

Intelligence is a word with a somewhat unclear meaning to begin with, but you have move the goalposts pretty damn far to exclude LLMs at this point. They are certainly still lacking in some regards, but whether that disqualifies them for intelligence is very much a matter of debate.

[nadir_ishiguro]

"And I'm sorry, but you're not up to date about interpretability literature, or for that matter philosophical discourse,"

I see no citations referring to the current philsophical discourse, unless you mean to imply anthropic's paid people are to be considered to be part of that.

That they "see structures in Claude akin to those we think are associated with human consciousnesses experience in the brain" is if anything discrediting.

[layla5alive]

I guess you don't really know how an LLM works then..?

[0xDEAFBEAD]

In this very thread I am being told that Fable is nothing but a bit of scale and refinement on well-known neural network techniques. And next I am told that we can't even imagine how to build superintelligence. Which is it folks?

[layla5alive]

Indeed, the real fun signal here is the conflict between how intelligent humans think they are and the different story told by much of human behavior, including within this very thread!

When an LLM makes a cognitive mistake, humans jeer "dumb stochastic parrot!" - when a human does the same, many are blind to it. Like a weird auto-Gell-Mann Amnesia - relative to reflecting on our own perceived cognitive "strengths" versus others weaknesses.

[victorbjorklund]

It’s also theoretically possible to travel almost at the speed of light. Doesn’t mean it’s rational to talk about it today as if imminent.

[lins1909]

How can not believing something that hasn't been proven be unscientific? Do you know that words mean things?

[ToValueFunfetti]

Science means the pursuit of knowledge. It doesn't mean "only believing proven things". If we're going to be rude, lets at least take the time to be right.

[trebaud]

You dont seem to understand what proof means. Human brain is made of matter, matter can be arranged to make a thing that reproduces human brain properties. What's the confusion here? I say its unscientific because it places the human brain beyond the scope of what can be operated on. Not having the knowledge or tech yet to achieve that is irrelevant since we have an existence proof.

[TheOtherHobbes]

Even if we accept your premise, and not everyone does, the confusion is whether we're capable of creating an equivalent arrangement, even in principle, using alternative materials.

That's not "irrelevant", it's fundamental.

[JumpCrisscross]

> Not believing that AGI is possible

One can simultaneously believe AGI is possible, be only modestly sceptical that our current methods are likely to yield it in the near term and still find the religious ferocity enveloping its discussion silly.

> saying that the human brain cognitive capabilities cannot be reproduced on other types of substrates is stupid at this point

Straw man. Nobody argued this. The discussion is around how urgent it is to policy treat a future hypothetical.

[ToValueFunfetti]

Nobody argued anything. GP just dismissed it as religion without engaging with a word of the material. Parent is taking a stab at why.

Not that I'm complaining. Cynicism is the failure mode I rely on HN for. It's the populism that's been getting to me.

[JumpCrisscross]

> GP just dismissed it as religion without engaging with a word of the material

Fair enough. I didn’t see anything novel in the article. So treating it as a motif within the abovequoted “Superintelligence: The Idea That Eats Smart People” context is fair and a real argument.

> Cynicism

Cynicism isn’t the opposite of blind optimism. Nihilism is. I’m not seeing a rejection of the article as being baseless as cynical or nihilist. It’s just pointing out a cultural thread that doesn’t seem to be useful.

[ToValueFunfetti]

I hate to get bogged down in semantics, but with the hope that one of the stronger top-level critiques makes it into the top slot here and this conversation gets buried:

Cynicism is defined as

>An attitude of scornful or jaded negativity, especially a general distrust of the integrity or professed motives of others.

I'm not saying that cynicism is automatically wrong, just that I once could trust that, when HN is wrong, it is due to cynicism applied in excess.

[oliculipolicula]

>..Nothing in life ... has any chance of succeeding without strenuous practice; and this is capable of overcoming anything..

---Diogenes (L) on Diogenes, arch-Cynic, VI.2.71

Afaik not formalised in Lean4, but sounds like optimism

Most of D's other stuff imho is optimism taken to its clear-eyed extreme. ya know. the way a very best version of us could always unironically RELY on HN [,after much practice]

>so those whose training has been of the opposite kind derive more pleasure from despising pleasure than from the pleasures themselves.

Might be why professional Nihilists admire Cynics (ie, for their ... unaffected comedy)

https://penelope.uchicago.edu/Thayer/E/Roman/Texts/Diogenes_...

[ToValueFunfetti]

Ah, this clarifies things. I suppose I brought this on myself by putting it at the start of the sentence, but I do mean little-c cynicism, not the ancient philosophy. I was surprised to be called out on it, as GP is a shining example, and I was confused by the mention of nihilism, but I see now where wires were crossed. Thanks!

[threetonesun]

I put it in the same bucket as living on Mars. Can it be done? Probably. Are we close? Not as close as people seem to want to believe. Is it a goal that will largely benefit society in its current form? Absolutely not.

[JumpCrisscross]

Eh, with Martian habitation we know what the roadmap looks like. With AGI we don’t. It could be proximate. Or it might not be. When it arrives, it could be totally economically uncompetitive outside the rich world. Or it could replace all human labour. Or progress to become a superintelligence.

We don’t know. Which makes proposing rules around it based on fiction more than science silly.

[neutronicus]

In particular, I just don’t buy into the “left behind unless” framework.

Perhaps Anthropic will create God in the Machine. Not foreclosing on that. But will it matter so much who was fucking around with Opus five e-folding times ago?

Either ClauDeus is benevolent and lifts you up (not left behind) or it isn’t, or not to you, and you are culled by a drone (left behind regardless).

Serenity Prayer time.

[gilrain]

> Not believing that AGI is possible is irrational and unscientific imo

This is an objectively wrong opinion.