[rrmm]

One thing I've been considering: At what point does a creator have a moral or ethical obligation to a creation. Say you create an AI in a virtual world that keeps track of some sense of discomfort. How complex does the AI have to get to require some obligation? Just enough complexity to exhibit distress in a way to stir the creator's sympathy or empathy?

The glib answer is never, of course. And one easy-out, I can think of is setting a fixed/limited lifespan for the AI and maybe allow suicide or an off-button. So the AI can ultimately choose to 'opt-out' should it like; and at least, suffering isn't infinite or unending.

It reminds me of reactions to testing the stability of Boston Dynamic's early pack animal. The people giving the demo were basically kicking it, while the machine struggled to maintain its balance. The machine didn't have the capacity to care, but to a person viewing it, it looked exactly like an animal in distress.

[dqpb]

> The glib answer is never, of course

Dismissing “never” offhand without explanation is glib.

[OscarCunningham]

Utility functions are only defined up to addition of a constant and scaling by a positive constant. So instead of rewarding them with +5 and punishing them with -5, you can use 1005 and 995 instead. Problem solved.

[rrmm]

The numbers are indeed arbitrary. But ultimately you want to avoid low utility/reward action and continue high utility/reward actions. That behavior, trying to avoid or pursue actions, would be indicative of the state of distress regardless of an arbitrary number attached to it.