[visarga]

> they're in essence, the last company to ever exist, and are building the last product we'll ever need

Physical reality is the ultimate rate-limiter. You can train on all of humanity's past experiences, but you can't parallelize new discoveries the same way.

Think about why we still run physical experiments in science. Even with our most advanced simulation capabilities, we need to actually build the fusion reactor, test the drug molecule, or observe the distant galaxy. Each of these requires stepping into genuinely unknown territory where your training data ends.

The bottleneck isn't computational - it's experimental. No matter how powerful your AGI becomes, it still has to interact with reality sequentially. You can't parallelize reality itself. NASA can run millions of simulations of Mars missions, but ultimately needs to actually land rovers on Mars to make real discoveries.

This is why the "last company" thesis breaks down. Knowledge of the past can be centralized, but exploration of the future is inherently distributed and social. Even if you built the most powerful AGI system imaginable, it would still benefit from having millions of sensors, experiments, and interaction points running in parallel across the world.

It's the difference between having a really good map vs. actually exploring new territory. The map can be centralized and copied infinitely. But new exploration is bounded by physics and time.

[Teever]

To conquer the physical world the idea of AGI must merge with the idea of a self replicating machine.

The magnum opus of this notion is the Von Neumann probe.

With the entire galaxy and eventually universe to run these experiments the map will become as close to the territory as it can.

[visarga]

Fully agree, self replication is key. But we can't automate GPU production yet.

Current GPU manufacturing is probably one of the most complex human endeavors we've ever created. You need incredibly precise photolithography, ultra-pure materials, clean rooms, specialized equipment that itself requires other specialized equipment to make... It's this massive tree of interdependent technologies and processes.

This supply chain can only exist if it is economically viable, so it needs large demand to pay for the cost of development. Plus you need the accumulated knowledge and skills of millions of educated workers - engineers, scientists, technicians, operators - who themselves require schools, universities, research institutions. And those people need functioning societies with healthcare, food production, infrastructure...

Getting an AI to replicate autonomously would be like asking it to bootstrap modern economy from scratch.

[Teever]

I think that we're going to approach it from the top and bottom.

The second we have humanoid robots that can do maintenance on themselves as well as operate their assembly lines and assembly lines in general will be a massive shift.

I think the baseline for that will be a humanoid robot that has the price tag of a luxury car and that can load/unload the dishwasher as well as load/unload the washing machine/dryer and fold and put away clothes. That will be total boomer-bait for people who want to 'age in place' and long term care homes in general.

Once we have that we can focus on self-replication on the micro-scale. There is tremendous prior art in the form of ribosomes and cells in general. A single cell hundreds of millions of years ago was able to completely reshape the entire face of the earth and create every single organism that has come and gone on the Earth. From fungi to great whales to giraffes, jellyfish, flying squirrels, and sequoia trees the incredible variety of proteins in a myriad of configurations that life has produced is remarkable.

If we can harness that sort of self replication to make power our economy it will make the idea of bootstrapping the economy on this world and others much easier.

[pixelsort]

It seems that anyone who has ever played games like Factorio or Satisfactory can readily extrapolate similar real-world conclusions. Physical inefficiencies are merely an interface issue that erodes over time with intelligent modularizations and staging of form factors at various scales.

[optimalsolver]

This might come as a surprise to some people, but the real world is infinitely more complex than a sim game.