[TheOtherHobbes]

That's more worrying than silicon AGI, which I think is a non-starter.

The basic problem with AGI is that software is ridiculously brittle and ad hoc, with no useful heuristics for general - as opposed to task-oriented - self improvement.

We can't even make bug-free web pages. So the idea that we can engineer a bug-free self-improving AGI is unconvincing.

Moving to neo-biology doesn't necessarily change that - but it does mean we could end up with buggy systems that chase you around and eat you, as opposed to crashing your ad blocker.

[pdimitar]

> We can't even make bug-free web pages. So the idea that we can engineer a bug-free self-improving AGI is unconvincing.

This is mostly true because of current economical realities in the IT sector.

If very rich people have long-term plans to invent an AGI and they have labs where trying to do so is an everyday activity then I am pretty sure they'll use more serious methodologies.

So what you say is true but only within the bounds of the lowest common denominator ("a regular programmer working a wage job").

[nradov]

Even software developed with effectively unlimited resources is rather buggy. The closest humanity has ever come to truly reliable software was probably the Space Shuttle flight control system, and that code was relatively simple in a highly constrained problem domain. There is zero evidence that "serious methodologies" would get us any closer to a working AGI.

[pdimitar]

Oh, definitely. I am not saying that not doing the normal agile crap for development brings us closer to AGI. What I would venture to say though is that trying stuff like formal verification and finding a way to integrate it in a daily workflow would definitely help with the average quality of software developed through such a process. And that maybe the private labs are more liberal and let people's creativity achieve results.

Admittedly, yes, there's no proof. But we as an area are pretty stuck lately IMO. But that goes off-topic.