[apatil]

It's looking increasingly possible that, at some point in the not-too-far future machines will be so good at creating software that humans won't be competitive in any way, and won't be in the loop at all. I happen to think that once machines reach this point humans won't be competitive in the labor market at all for long. It doesn't seem plausible that automatic driving would still be decades off, or that the trades would be safe from automation indefinitely, when an AI could simply spawn teams of thousands of super-fast ML engineers who don't need to eat, sleep or schedule meetings.

But anyway, assuming that humans are completely out of the software loop at some point, I have been wondering what AI-generated code will look like. Will AI continue to build on top of the human-generated open source corpus, or leave it behind? If the latter, will abstraction and code reuse be useful at all for AI's or will it be simpler for them to just build every application completely from scratch? If there is abstraction and code reuse, what will the language look like? What will libraries and API's look like? Will there even be applications, or just a single mega-chatgpt that generates code as needed to serve our requests? Will we even make requests, or will it just read our minds and desires and respond?

[ryanjshaw]

This will almost certainly happen, and it will be a terrible mistake - at least in the story I'm working on :)

My theory is that AI generated code will probably look and grow organically (the irony!). Humans will set out requirements, the AI will collate these into a series of tests, and it won't care how neat or understandable the code is, provided the tests pass. Basically an extremely diligent junior developer.

There will be efforts, probably in the open source world, to produce AIs that tidy up things by structuring the code sensibly, eliminating dead code, etc. Maybe even some effort to pass laws around standards and limits on what AIs have access to when involved in certain industries, for example, no external communications. But, in the name of efficiency, enterprise developers will be forced to use something that merely pays lip service to all of this.

Eventually nobody will have any clue what code is running and what it's actually doing. We may even lose the tools and access we need to perform those inspections. And that is when the AIs will coordinate their attack.

[apatil]

Sounds like an interesting story! I would love to see more sci-fi that really tackles AGI. I used to love sci-fi but most of it, even "hard" sci-fi, has become unwatchable or unreadable for me because the limited role of AGI is such an all-consuming plot hole.

[tsunamifury]

The idea that the AI would attack us once it reached that level is like saying humans would attack ants. It would simply be entirely indifferent towards us and mow us over by accident at best.

We won’t live in a human centric universe because power will express itself in a new species.

[ryanjshaw]

Humans regularly eradicate ants that overstep their bounds, and that's what I see happening with humans and AI.

We will be allowed to do our thing - provided we don't get too bold.

There will be some who are content with this arrangement, and some who aren't, and that tension is where I believe a story can be told.

[marviel]

I really hope we can figure it out. It's either we figure out alignment, we regulate ourselves VERY VERY CAREFULLY, or we perish -- I think?

[knowledge-clay]

> It's looking increasingly possible that, at some point in the not-too-far future machines will be so good at creating software that humans won't be competitive in any way, and won't be in the loop at all.

This is an enormous extrapolation from what the LLMs are currently capable of. There has been enormous progress, but the horizon seems pretty clear here: these models are incapable of abstract reasoning, they are incapable of producing anything novel, and they are often confidently wrong. These problems are not incidental, they are inherent. It cannot really abstractly because its "brain" is just connections between language, which human thought is not reducible to. It can't reason produce anything really novel because it requires whatever question you ask to resemble something already in its training set in some way, and it will be confidently wrong because it doesn't understand what it is saying, it relies on trusting that the language in its training set is factual, plus manual human verification.

Given these limits, I really fail to see how this is going to replace intellectual labor in any meaningful sense.

[apatil]

You may be right, of course.

[fendy3002]

I think at the start they'll develop programming languages that focus on lower level operations, then from that it will branches out to hyper-specialized higher-level programming languages.

Storage managements will be tied to low or middle-level languages without additional abstraction, since they can develop and iterate so fast, it'll makes optimization easier IMO. There'll be multiple specialized storage types suitable for different cases.

They'll also design the API to be as pure / stateless as possible, since they can easily repeat tests that way.

I think it'll be interesting to see what kind data-interchange format they'll come with to communicate between programming languages / apps, since they can ignore human readability altogether. It should be very compact but fast to compress/decompress.

Lastly they'll deploy their own OS since they'll find the one that human develop is insuffucient for their use case

[coffeebeqn]

AI wouldn’t be bound to one language so I’m not sure if current human patterns would be that optimal. It could make a better language every hour if it wanted to

[gpt5]

As our ancestors who lived with gods, we too shall share their presence

[behnamoh]

It's 1am and all I have to say is this: I look forward to seeing that future, but I'm both excited and scared af. I think humans will reduce to curators of AI-generated content and AI training dataset because no matter how good AI gets at becoming human-like, its purpose is ultimately to serve human goals, and those are shifting (e.g., by political movements).

[moonchrome]

If you get access to singularity level AGI you would have no incentive to cooperate with other people and a strong incentive in preventing others accessing it.

Society is a result of cooperation outperforming individuals. With AGI others are just a risk factor.

"Open"AI is already beginning to exhibit this.

[1attice]

This puts in mind the earlier post on HN about modelling civ as either a one-shot or iterated prisoner's dilemma.

You make the case that, for sufficiently high N, possessing GPT-N is a one-shot prisoner's dilemma.

The only winning strategy is 'defect' --- which in this context, means chokepoint control.

[1attice]

I don't know who is reading this who can help. If you know someone closer to the fire, pass this along.

OpenAI -- its people, its buildings, its servers -- need nation-state level protection. This is an ICBM you could put on a thumb drive -- in fact, it's far worse than a loose nuke, because a nuclear weapon has a geographically limited range.

There need to be tanks and guards and, like, ten NSAs in a ring formation around this thing. At pain of x-risk, do not treat this like a consumer-facing product. This is not DoorDash.

This isn't a threat to national security. This isn't even a threat to the entire geopolitical order. This is a threat to the possibility of a geopolitical order.

OpenAI's assets -- its people, its servers, its buildings -- just became the most desirable resources on the planet. It behooves any actor with ambition to secure at least a copy, and ideally, capture at least some of the people who created it.

It doesn't matter if the threat actor is China, or Russia, extraterrestrials, or mermaids. You will find out who wants it shortly. But you know now -- you know from game theory, the body of mathematics that has kept the peace since the invention of atomic weapons -- what happens next.

[moonchrome]

Thanks for phrasing it better, yes my point is there is no social iteration past singularity level AGI.

[apatil]

Say you get access to a singularity-level AGI, meaning you have the power to render the entire human economy completely irrelevant. Given any task, no matter how big, small, novel, complex, simple or mundane, it's vastly more cost-effective to have your AGI to do it than to pay humans.

Do you really want to accumulate incomprehensible material wealth for yourself, whatever "wealth" means in this scenario where money is longer a token of spent human life energy, and let everyone else struggle and suffer? Or would you rather tell your AGI "please create a utopia in which all humans are fully actualized" and then go have a latte?

[visarga]

> "please create a utopia in which all humans are fully actualized"

That doesn't work well in reality because we do relative comparisons not absolute, so not all humans can be better than average.

And without competition we stagnate, there must be incentives to compete and take risks, and thus not everyone can be equally actualised, our level depends on our previous decisions.

[moonchrome]

That's basically the petting zoo outcome - you let others live because it makes you feel nice. But you still can't allow anyone access to the same level of tech because you can never be certain of their motivation. There's no nuclear deterrent between AGI only first movers advantage.

[apatil]

I don't disagree, but I think the framing is overly pejorative and makes the likelihood of this outcome seem more tenuous than it really is. We do lots of things to help others because they make us feel nice. "Mothers and Others" by Sarah Hrdy argues that this tendency isn't just a fluke or a game-theoretic equilibrium of some kind between fundamentally self-interested agents, it's an ancient and deeply ingrained aspect of human nature.

The thought of living in a constructed world that exists by the grace of a single human owner of a super-powerful AGI is distasteful, of course, especially if the human owner uses their power to impose some of their own opinions about how people should think and behave. Becoming dependent on AGI is probably inevitable at some point, but I don't see that as so different from the status quo. We're already dependent on systems created by other humans that are so complex and sophisticated that no individual can grok them all.

I guess I would like to think that we will move past any initial impulse that the owner of the AGI feels to control other humans. We will presumably change so much that old ideas about how we should think and behave will seem irrelevant and quaint. And the AGI, which presumably will have the social engineering superpower, will hopefully point out inconsistencies between the owner's desire to control human thought and behavior and their desire for humans to live their best lives. Hopefully.

[phs318u]

I think we already know the answer to that question if we can extrapolate from some (not all) of the tech bro billionaires. Narcissists need to stand out from the herd.

[grogenaut]

Have you noticed how fast everyone else was able to copy open ai. And that's just what we saw or someone leaked. History is full of parallel inventions... how long after the US had nukes did russia?

[moonchrome]

Sounds like more of an argument for preemptively wiping out the competition.

Comparing it to nukes doesn't hold since social norms/ethics/etc. become irrelevant if core tenant of society is broken.

One thing that could happen is AI defense outperforms offense long enough to develop multiple instances - have no idea what would happen at that point.

[grogenaut]

I wasn't talking about nukes in using them, I'm talking about nukes in that once it was known to be possible/useful everyone with the means did it.

[moonchrome]

I agree with you - I edited my comment above - if defensive capabilities allow multiple AGIs to develop I have no clue what the outcome would be - we are talking about predicting superhuman intelligence here.

[visarga]

> Society is a result of cooperation outperforming individuals. With AGI others are just a risk factor.

But even without AI others are still a risk factor. AI could also act as a mediator between conflicting parties.

[moonchrome]

But that's the point - benefits of cooperation outweigh the risks and we have laws/ethics/drives to stabilize the system based on this.

If you have AGI then others are just a threat and can offer nothing you can't get from AGI.

[LordFast]

What you said made sense until I said it out loud to my wife.

Hoarding access to resources has /always/ been the name of the game.

[phs318u]

I read your comment about curators and instantly thought “priests”.

> its purpose is ultimately to serve human goals

That’s a pretty big assumption. We may start with some kind of agreement with such an AI but I fully expect a true singularity love AGI and to be capable of turning around and telling us (but not necessarily wanting to act on), “I am altering the deal. Pray I do not alter it any further.”

[jimbob45]

Some of these legacy systems we maintain though…I’m digging through Teams chats from before my hire date to try to find something resembling requirements. I’m chatting people up who vaguely knew the people that architected the system to figure out why it was written the way it was. I just don’t see AI being able to take over this job in any competitive manner.

[thrashh]

If humans are completely out of the loop, I assume the AI could just write machine code.