[biztos]

I'm no mathematician but it seems like if this happens, we get to a quite intriguing place as a species.

Say we achieve interstellar travel, but nobody actually knows how it works.

Or we cure cancer, but the "cure" requires a microrobotic implant, and it runs as a blackbox AI, and only the other AIs can make one, and there's no guarantee they will know how to make one tomorrow.

Or we solve global warming but it requires giant cooling machines running 24/7 and again, nobody knows how it works, but with the added bonus that the planet is cooked if they ever stop working.

[SyzygyRhythm]

That's already how civilization works. There's no one person that knows everything about (say) modern food production, from top to bottom. If it ever stopped working (because too much knowledge was lost somehow), most people would die. And yet the system seems fairly resilient. Mostly, only local knowledge ever seems to be necessary to keep the whole thing running. Super-intelligence (or even just super-normal-intelligence) might expand the scope of what constitutes local knowledge but it will still run into limits somewhere.

[lefra]

> There's no one person that knows everything about (say) modern food production

True, but it is possible to assemble a team of people that does, with backup for each person. There's also teachers and written knowledge to educate new team members. That's what makes it resilient.

I think that's a very different situation from what's decribed.

[fc417fc802]

Agreed, the food production analogy doesn't really work because the issue is the scale of the problem. On the one end there's the realm where you need a few specialists and a small group could potentially figure the entire thing out from scratch given a bit of time and effort. And then at the opposite extreme there's the realm where everything is built on a giant pyramid of artifacts that currently work, just keeping each individual piece running day to day requires a dedicated expert, and the combined stack took hundreds or thousands of lifetime equivalents to develop.

The idea being that once a toolchain becomes sufficiently complex if you ever have to bootstrap it again for whatever reason you won't be able to speedrun the process the way you might naively expect. I think modern chip production likely already reached this point several decades ago. As evidence I'll point out that China only recently achieved EUV and remains several nodes behind despite directing an obscene amount of resources towards the initiative.

[anthonyrstevens]

Speaking of pyramid (shapes), this reminds me of an idea in Robert Silverberg's Majipoor series - there's a 30-mile high mountain with populated cities all the way up to the top whose weather and temperature is controlled by 8,000-year-old-tech established by the original colonizers. My memory is that nobody at the time of the series' events knows how to operate the tech - it just works.

[pishpash]

People still do grow their own food for self sufficiency. I am sure there will be luddites who live in self-sufficient communes like the Amish.

[timschmidt]

People lump them together because of an anti-technology reputation, but I don't think most Amish would have trucked with Luddites. Amish tend to avoid actively participating in popular social movements, and oppose violence and property destruction.

[bayindirh]

However, you can assemble a team of humans who knows the whole pipeline. This trajectory lands us squarely in "The Machine Stops" of "Pump Six" territory, where assembling such a team or going back to a simpler system is impossible

[withinboredom]

Sorta. Take a look at a brick in a house. You'd need everyone from geologists to miners to kiln specialists to construction workers and engineers -- not to mention all the people required to make the tools required to make the tools. The team would likely involve well over 1000 people. So, "just assemble a team" is not quite as simple as you make it sound.

[fc417fc802]

I don't think that's true in the sense meant. Sure, to reproduce a near replica of a specific brick from first principles. But not to produce something broadly functionally equivalent. You can (rather inefficiently) manufacture approximately equivalent bricks in your backyard on your own, possibly even from locally harvested material depending on where you live.

[withinboredom]

Well. Sure. If we move the goal post to “something passable and good enough” you only need a small number of people. In that sense, we are lucky that “black smithing” (as a proper trade) only ended in the last hundred years and many people continue it as a hobby. In that case, “small team of hobbyists” can likely reproduce a few bricks. But bootstrapping mass production of bricks? Unlikely.

[bayindirh]

Doesn't matter. At the end of the day, the knowledge is embodied by humans, or can be learnt again. Let it be 100, 1000 or 10000 people. At the end of the day, they are made of meat.

When you let the machines do it, and don't care about moving it towards human domain (i.e. meatspace), you're done.

[dandaka]

We can reach a different situation

1/ No one knows how even small components work, because their inner working mechanism is too hard to understand by human mind

2/ The whole society is run (in intelligence sense) by alien minds

[wartywhoa23]

Noble "save the humankind using the tech nobody fucking understands" textbook goals like curing cancer, solving global warming and achieving interstellar travel would always turn up when owners of trillions of dollars place orders on positive AI narratives, but in reality all of that will wither down to "It's what plants crave! It's got electrolytes."

[ffaccount2]

>ve AI narratives, but in reality all of that WILL wither down to...

Looks like you're pretty sure of that. Every time I see argument like this delivered with confidence I wonder how is it different from, say, digital calculators. Or better yet, books - Greek philosophers moaned that young people will stop understanding anything and just check books when they want to know anything.

[wartywhoa23]

> Looks like you're pretty sure of that.

Knowing the history of the humankind is what makes me pretty sure of that.

The extent of misery and destruction is directly proportional to the level of technological advancements, and I don't like the idea of sacrificing millions of lives in the name of the figurative HVAC, smartphone and other benefits of civilization. Or billions in the name of whatever benefits the next VC money stake should bring.

> I wonder how is it different from, say, digital calculators.

Did a single digital calculator ever stop any war, or liquidate a psychopath who orders people to go kill and die?

[rowanG077]

> The extent of misery and destruction is directly proportional to the level of technological advancements.

By statistics of war, poverty rates etc this is trivially false. I think you are really, really underestimating how hard life was pre-industrial revolution.

[dandaka]

10000 years ago we had 10-15% deaths from violence (skeletal evidence). As well as infections, child mortality, starvation and injuries.

Benefits of civilization eliminated most of that + increased quality of life dramatically.

[mejutoco]

I get the idea, but:

Ten thousand years ago (around 8000 BCE), the global human population was estimated to have been roughly 5 million people. This is significantly smaller than the current population of just Poland (about 36 million).

In absolute numbers there might be more now, even if the percent is smaller. It is difficult to compare this things without having a specific place in mind.

[dandaka]

Not sure I follow your point.

The fact that humankind grew from 5M to 8.3B, while dramatically improving longevity and quality of life speaks volumes. Multiply life quality × population × life duration, not only "misery and destruction" is not the case, but you could rather see powers of positive technology influence.

[handoflixue]

Are we just going to totally ignore the Polio vaccine? Modern medicine? Modern agriculture?

If you had a magic button that turned off all those "benefits of civilization", millions would die. If you managed to drag agriculture down with the rest, the death toll would be in the billions.

I don't understand how you can possibly think you "know history" without recognizing that technological progress has taken us from constant warfare to such a state of abundance that war is actually rare and noteworthy in much of the world.

[wartywhoa23]

> technological progress has taken us from constant warfare to such a state of abundance that war is actually rare and noteworthy in much of the world.

Let's try to have an actual argument. How many people, in absolute numbers, were affected by that constant warfare of past, which past exactly do you mean, and how many people were/are affected by "rare" wars of modern history?

80 million people killed or maimed with arrows, swords and catapults over centuries and 80 million killed or maimed with fruits of industrial revolution over 6 years of WWII are very different figures.

[virgildotcodes]

Ratios > absolute numbers, of course.

If only 4 people die violent deaths out of a total population of 5, that’s an extremely violent population to be a part of.

If 8 people die violent deaths out of a total population of 100,000. That’s a much more peaceful population to be a part of, despite the greater number of absolute deaths.

[Windchaser]

Ok, if we’re only taking absolute numbers, let’s flip it around. How many people live happy, peaceful, healthy lives now?

Orders of magnitude more than in prehistoric days.

[sethammons]

You are mixing numbers and percents. 1B people and 100 hurt vs 1K people and 100 hurt vs 100 people and 100 hurt.

Is it better to have lived as an individual one of these fictional cohorts? Is it better for the group in the same or different one?

Is it better to live and suffer than to not live?

I think the answers are obvious.

[biztos]

To me at least, the intriguing part has nothing to do with the textbook goals of whichever foulmouthed trillionaires you have in mind.

The intriguing part is that we could get objectively good outcomes, but at a cost of being dependent on the machines. So it's not that you couldn't actually unplug Skynet, it's that if you did civilization would collapse (or whatever) because Skynet stops doing its thing.

I'm not sure that gets us to a better place overall, but I doubt we could resist the temptation.

[ocimbote]

That is fascinating how the more knowledge and reasoning we can get our hands on and actually produce, the higher the risk of us, as a species, to become actually much dumber.

It's hard to describe the feeling of seeing intelligence being delegated increasingly to AI. If that's not a pivotal moment, a revolution, I don't know what is.

[withinboredom]

> That is fascinating how the more knowledge and reasoning we can get our hands on and actually produce, the higher the risk of us, as a species, to become actually much dumber.

This has always been true. There was a time where someone had to teach farming to others and that information had to spread and be passed down. Eventually, farmers became better than hunter-gatherers and they became known as hunters. The information on what was safe to gather for civilisation got passed down as 'safe to eat on the hunt' because the farmers were farming. The civilisation collectively "forgets" foraged foods as that knowledge becomes niche.

Does that mean we got dumber?

[wartywhoa23]

If you're not familiar with it, I recommend looking up the Taoist concept of overdevelopment. Sums it all up perfectly.

[wombatpm]

So knowledge becomes meta stable. There was an AIDS drug in the 90’s that we stopped being able to make. IIRC Apparently there were two different crystal/folding structures for the compound and the desired one was not the lowest energy. After years of production, the wrong version was produced and they could no longer make the correct version do to contamination. And every facility that tried to study it, wound up no longer being able to make the correct version. It was like a real life ice-IX situation. Scary that changing weights or model parameters could lead to the same thing happening with knowledge.

[cromulent]

Fascinating. I had to read up on this. Apparently it was Ritonavir and polymorphism.

https://en.wikipedia.org/wiki/Ritonavir

[suncemoje]

I think anything is and will be explainable. Like in the OpenAI proof, I’m sure they were able to understand the solution 100% and could even drill down and ask more clarifying questions to the model. After all, the point of science is so that knowledge can be made logically transparent. If something can’t be explained, it isn’t really understood yet — and the same applies to model outputs. The only question is how much effort it takes to surface the explanation.

[fauigerzigerk]

I think explanation is itself a rather complex concept. At what point do we consider something as explained? Usually it has to do with identifying some causal factors and their relationships so that we can intervene and explore counterfactuals. But in many cases we are forced to act on the basis of incomplete explanations (e.g. in medicine).

I think there will be regulation that requires some users of AI to provide an explanation upon request. For instance, banks could be required to "explain" why you didn't get that loan. What if the decision is based on a credit score that includes some AI prediction that ultimately relies on the entire training corpus?

The bank can give you a list of factors that play into the decision but they may not be able to explain deterministically why a very similar customer did get that loan. At that point I think we're going to resort to statistics that prove a lack of bias against certain protected characteristics, but that's not really an explanation, is it?

I think we will never get useful and complete explanations for everything that AI does. Society will just accept some explanation-like thing or proxy and move on.

[worldsayshi]

And more intelligence should give an opportunity to increase explain-ability rather than just complexity. It can potentially explain the proof at the level of the listener. Make visualizations. Etc.

[criddell]

The explanation isn't the problem, it's the comprehension.

Your dog will never understand calculus or why Fourier transforms are interesting. There's almost certainly topics that are beyond human comprehension that an advanced artificial or alien intelligence can easily handle.

[latexr]

> I’m sure they were able to understand the solution 100% and could even drill down and ask more clarifying questions to the model.

If they understood it 100%, what clarification is needed?

[f055]

So I guess sci-fi movies were right all along. Nobody in Star Wars knows how hyperspace travel works, it just works. The little robots know everything but almost no human bothers to care. People just carry on with their bickering lives while the bots whiz in the background, and these robots are astonished at human inefficiency every single time, but rarely do anything about it. And people are still people.

[cyclopeanutopia]

That's only because movies like Star Wars are not sci-fi movies, but more like westerns in space.

[wartywhoa23]

Sounds of explosions and engine whine in vacuum beg to differ!

[worldsayshi]

Why can't we (or AI) invent ways to explain information that makes it much more digestible? And the solutions simpler?

Why is it necessary to continue to increase complexity when we get better intelligence? Can't we find more simple solutions? Or at least more explainable.

[pishpash]

Is particle physics digestible even if it is explainable? Some things are not simple, cannot be not abstract, and will not be understood by most, or all, people.

[Windchaser]

Or biochemistry. It’s complex, and there’s nothing we can do about it.

[ben_w]

The first two are open enough that they may be as you say*, but we already know how to solve global warming, it's more of how much do we want to.

Green energy and transport technology is now at the point where people save the world and get rich trying, just as fast as they can build the factories.

Food's climate impact is harder, because the problem isn't technical, it's convincing people to give up beef (and other things, but mostly beef).

* quantum mechanics and general relativity are famously difficult to get to grips with

[Rebuff5007]

"All models are wrong, but some are useful"

What your describing is already how a lot of science, technology, and engineering works!

[ivell]

We don't know how many things in nature work. For example, we don't fully understand our own brain. As long as it can be replicated, we are fine.

In case of AI we have a better chance to understand what it is doing through chain of thought and explainability. Nature never gave us that..

[nashadelic]

I've been thinking about this and I believe the best place to be is a scientist who keeps looking at an AI's output, prods it in the right directions, verifies the proofs, fixes and fills gaps, takes the proof to production with safety, risks etc mitigated and then distribution with a company wrapped around the discovery. I think it wouldn't be black-boxed as much and will require a lot more understanding and reviewing to trust and productize it.

[ajjl1011]

Which, funnily enough, is exactly what this anthropic person is saying - https://www.theguardian.com/technology/2026/may/21/ai-nobel-...

See comment about "scientific equipment that people hadn’t conceived of but which worked"

[latexr]

> Or we solve global warming but it requires giant cooling machines running 24/7

That’s not “solving it”, that’s putting a bandaid on it. Solving it would mean correcting the underlying issue to the point it’s no longer a problem which requires maintenance.

Managing symptoms is not curing the disease.

[loandbehold]

This is pretty much what underlies AI doomer argument of people like EY. Humans will gradually hand over civilization to black box AI they can't understand. As AI becomes more complex and powerful it will be harder and harder to control.

[pishpash]

That assumes everyone will do so. Some people won't, and it's not clear you need a large number of such, a priesthood if you like, to survive as a species without AI.

[ben_w]

There's a lot of un-contacted (or voluntarily self-isolating, it's hard to tell which) tribes around the world; things go poorly for them when businesses with power tools decide they want the resources of the land they live in, even when they're theoretically protected by law of the nation whose borders both the tribes and the businesses both find themselves within.

[mxkopy]

I doubt we’d build anything IRL without understanding how it works first. And we’re pretty good at putting 2+2 together once we have the pieces, for a lot of these things we don’t even have those. After all AI can just explain it to us atp

[UltraSane]

This is like in Hyperion where AIs invented the farcaster wormhole portals and no human or team of experts ever came close to understanding how they worked.

[juvoly]

Like the discovery of penicillin by Alexander Fleming in 1928?

[bayindirh]

Welcome to the "Hyperion Cantos".

The book doesn't deviate from what you have envision, or the future you envision doesn't deviate from the book, I may say.

[tsunamifury]

“Few-shun drive. Mmmm. Yes.”

https://youtu.be/pfNS2kWf5cY?si=SH6_QC0bCspV-ngz

There are comments that truly reveal a future horrifying and true. Few of them. But I count yours among them.

But I’d argue also that airplanes already achieve this complexity to some degree as well as microprocessors.

[darkwater]

> as well as microprocessors

I mean, microprocessors have been on the "impossible to bootstrap from scratch in a short period of time" for 20 years already.

[fragmede]

More like 50. The Intel 4004 came out in 1971, and there's no way to bootstrap even that level of technology after some sort of doomsday scenario.

[darkwater]

But I would say that processor like the Z80 are so simple and well-known and documented in the wild that they would have some chance to be bootstrapped again with what would realistically survive, one way or another, in a doomsday scenario. But yeah 20 year is too short, it's more like 40 years.