Hacker News new | ask | show | jobs
by exoesquitur 2767 days ago
What you say is true for silicon, but Turing complete molecular machinery is already out there, we just have yet to master its intricacies.

The nightmare AGI scenario likely involves molecular computing using a chemically optimal code-base and engineered lifeforms a version or two up from current DNA microcode.

This is most likely to be created by a human-AI collaboration, which will be innocuous enough in itself.... it's the creations at this level that have the potential to replace us. Would that nesecarily be bad? Or just an enhanced form of human evolution?
1 comments
TheOtherHobbes 2767 days ago
That's more worrying than silicon AGI, which I think is a non-starter.

The basic problem with AGI is that software is ridiculously brittle and ad hoc, with no useful heuristics for general - as opposed to task-oriented - self improvement.

We can't even make bug-free web pages. So the idea that we can engineer a bug-free self-improving AGI is unconvincing.

Moving to neo-biology doesn't necessarily change that - but it does mean we could end up with buggy systems that chase you around and eat you, as opposed to crashing your ad blocker.

link
pdimitar 2767 days ago
> We can't even make bug-free web pages. So the idea that we can engineer a bug-free self-improving AGI is unconvincing.

This is mostly true because of current economical realities in the IT sector.

If very rich people have long-term plans to invent an AGI and they have labs where trying to do so is an everyday activity then I am pretty sure they'll use more serious methodologies.

So what you say is true but only within the bounds of the lowest common denominator ("a regular programmer working a wage job").

link
nradov 2767 days ago
Even software developed with effectively unlimited resources is rather buggy. The closest humanity has ever come to truly reliable software was probably the Space Shuttle flight control system, and that code was relatively simple in a highly constrained problem domain. There is zero evidence that "serious methodologies" would get us any closer to a working AGI.
link
pdimitar 2767 days ago
Oh, definitely. I am not saying that not doing the normal agile crap for development brings us closer to AGI. What I would venture to say though is that trying stuff like formal verification and finding a way to integrate it in a daily workflow would definitely help with the average quality of software developed through such a process. And that maybe the private labs are more liberal and let people's creativity achieve results.

Admittedly, yes, there's no proof. But we as an area are pretty stuck lately IMO. But that goes off-topic.

link