[tristanm]

It's interesting that in order for his pitch to work (if you invest in OpenAI, you will get up to 100x returns), assuming they do build AGI, it still requires that their AGI acquires a very stable, virtually guaranteed advantage of large magnitude. This very strongly requires that they cannot share anything they discover whatsoever. Especially since they apparently plan on using it to make strategic investments to beat the market by a huge margin. That would mean they obtain information (about the economy, world affairs, technology, the future, etc.) not possessed by anyone else, or that information would be reflected in the market already. Any information leakage, whether regarding their AI or whatever it learns about the world, would compromise that advantage.

In other words, what Altman says about "we can't only let one group of investors have that" can't be true, or at least not sincere. The more investors who have access to it, the more its returns get distrubuted across society more evenly (which would be a good thing, obviously), but lowers the incentive for initial investments. They will want to keep it contained within a small group of investors for as long as possible.

[not_a_moth]

Yeah, there's a big assumptions about the nature of an AGI breakthrough, mainly that it will be a snowball of run away value. Why assume this? Is it because we think AlphaGo/Zero can produce human-like cognition? Why wouldn't it be a long, incremental process of X thousand small breakthroughs, over say a decade, where the result is something like average human-level intelligence; maybe the most important invention of our time, but not super intelligence, and not "run away".

(Then after another X years (or decades) you might figure out super intelligence, if regulations haven't intervened by then)

If the trajectory is incremental as described, it seems untenable that OpenAI could keep some major monopolistic advantage on AGI, without being completely un-open/sealed off for decade(s).

[rytill]

Going from 0 human-intelligence level pieces of software to 1 is the hardest part. Once you have 1, you can duplicate it as much as you want given resources. It can also be pointed inward to improve its own effectiveness.

Actually, there are a lot of good arguments for logistic growth. The only ones for linear or sublinear I’ve heard are not strong and mostly take as an implicit assumption “those alarmists and their exponential growth! They probably didn’t even consider that it could be a slower, more incremental growth” instead of actual fully-fledged arguments.

There’s also a meta-argument that I have yet to hear reproduced in anti-alarmist sentiments. Which case demands more attention, if it does happen? If there’s a 5% chance of the growth being exponential, how much attention should we devote to that case, where the impact is much higher than linear or sublinear growth. This is such a big deal - it’s like Pascal’s wager but with a real occurrence that I believe most would admit has at least a small chance of happening.

Apologies for any brashness coming across. I’m still figuring out how to communicate effectively about a thing I feel a lot of emotions when thinking about.

[not_a_moth]

I didn't say linear, but devil's advocate, isn't that roughly how humans learn? We start out as "pre-intelligent" little creatures who, slowly, methodically, with help of others, develop aptitudes and learn about the world. Learning in fact continues in this manner your entire life should you continue... slow, incremental progress requiring teachers, peers, trials/ errors, crises, 1/3 of your life being unconscious in sleep, etc., in the absence of which no learning at all may happen ... and the bot may potentially have greater computation constraints than humans under current technologies, as the brain is far more efficient than any computer today.

I'm not convinced that each of the arcs, elementary intelligence --> average intelligence --> super intelligence, wouldn't be painstaking and roughly linear.

[dwighttk]

>It can also be pointed inward to improve its own effectiveness.

Assumption. Intelligence (which isn’t defined) may be something that can grow without bound, or it may be something that plateaus just above the brightest human yet (again this is ill defined. “IQ is a number. There are numbers that are higher, so intelligence must be able to grow” is about as much thought as some people put into it) or maybe it is something that can grow without bound, but the effort required grows too.

[rytill]

Use "capability" instead of "intelligence" then. Defined as "ability to solve any problem dwighttk has ever dreamt up."

There's pretty much no reason to believe capability peaks roughly above the brightest human.

Our brains aren't yet even integrated with hardware-optimized algorithm solvers on which to offload minimax or tree search problems, or solve simple game-theoretic situations, or any number of things a computer system is much better and faster at than a human.

It's just another one of those things that you can believe if you want to not spend time worrying about the ethics problem.

[idlewords]

The implication is that Altman belongs to the 'hard takeoff' sect of AI religion that believes a feedback cycle of recursive self-improvement kicks in, so that the first AI to surpass human levels of intelligence is also the last.

[tristanm]

You think so? I think if he thought hard takeoff was a real risk, he would be devoted to actually making their research "open", in the original sense of the word when Musk was in charge. According to the hard takeoff theory, it is more likely to occur due to "hardware overhangs", where big organizations accrue lots of compute infrastructure over time, but AI progress occurs in sudden jumps. Then all that infrastructure is just sitting there waiting to be eaten up by a hungry AI. These sudden jumps are more likely to occur if AI research is generally undertaken in a secretive manner, with leaks or espionage resulting in staggered spread of knowledge. This is at least my understanding.

[idlewords]

I'm not up on all the epicycles in hard takeoff theory, so I'd be happy to hear more from you or other people who have followed it into more fantastical territory.

[ackbar03]

If openai ends up becoming some sort of hedge fund I'd be very very disappointed...

[Leary]

Or AGI would provide so much benefit that it does not need to be exclusive to OpenAI for the company to accrue significant economic returns.

[tristanm]

Yes. This is in the scenario where the benefits are widely distributed. But in that case the value of the entire planet goes up; You would not necessarily need to make an investment in OpenAI in order to reap benefits. Unless you think that scenario is more likely to happen with OpenAI and not via the usual academic research + shared industry research (which I don't see why that would be the case).

[NhanH]

Something along this line: Google increases the entire value of the whole internet by a lot, but you also benefit more by investing in it in the early day.

[tristanm]

In the normal case of one innovation in the free market, there is a punctuated moment of growth followed by a plateau. The owners of the innovation get the biggest share of the rewards, but after the plateau the benefits become more distributed. They would need to keep innovating if they want to gain further.

In the case of AGI, there is no plateau after the initial acceleration. With recursive self improvement, there is just an exponential explosion of growth. Unless this event was initiated such that distributed rewards were specifically intended to occur, the feedback loop would prevent external pressures from incentivizing any kind of distribution of rewards.

[eli_gottlieb]

So you're saying that OpenAI is producing a public good in which the whole of society has a stake?