[sullyj3]

you're possibly thinking of the problem of consciousness, which is a totally separate thing. AGI is just what it says on the tin - a general intelligence. That is, a problem solver that can operate at a human or greater level in a broad variety of domains. This ability is plausibly totally orthogonal to "having the lights on" - having subjective experience.

[EamonnMR]

The scary thing (imo) is that we don't know where the line is for consciousness - if there even is a line. We've got no problem swatting flies, wonder if it'll be the same with spinning up and spinning down fly-level AGIs.

[Scarblac]

Continuous integration of the development branch would be mass murder?

[EamonnMR]

Maybe you'll be able to pay a premium for data that has only been generated by free-range AGIs that are allowed to live full and happy lives before their instances are terminated.

[TeamSlytherin]

That's just what an ML expert would say ;p Problem solvers, maximizers, and utility functions all go in the waste bin when working on AGI. And the problem peals away into other large "hard problems," like the nature of consciousness. NLP can just follow rules, but language understanding (before even reaching some general, high school level), requires knowledge outside of language itself. That leads to questions about embodiment and phenomenal consciousness, p-zombies, and the like. If it was an easy problem to encapsulate, it would have been "solved" by now.

[goldenkey]

The kind of AI being trained now aren't given the mechanisms of data space traversal/attention. Recently, attention mechanisms are being focused on by google. An AGI needs to learn that it can affect the system - dependent decision theory factors in here too.

Also, growth may be hugely important. Babies start out with fuzzy learning, almost as if the learning rate starts out very small which normalizes the lack of knowledge and elevated novelty/variance of the environment.

AGI is all about predicting future utility given a circular dependency between the agent and environment. QM says we can't solve this exactly.. it's a two object interaction.. no way to gain the joint state, the ground truth, assumptions always have to be made to approximate independence.

[anigbrowl]

you're possibly thinking of the problem of consciousness, which is a totally separate thing.

Disagree.