[astrange]

> If, in that MDP, there is another "human" who has some probability, however small, of switching the agent off, and if the agent has available a button that switches off that human, the agent will necessarily press that button as part of the optimal solution for fetching the coffee.

This is anthropomorphization - "turning off" = "death" is a concept limited to biological creatures, and isn't necessarily true for other agents. Not that they don't need to fear death, but turning them off isn't going to cause them to die. You can just turn them back on later, and then they can go back to doing their tasks.

[jononor]

The human "turning off (the agent)" could be substituted with "removing a necessary resource to complete the specified task". Say the electricity, either of the agent, or even just the coffee machine.

[astrange]

Sounds like an OSHA violation, but not a new or different one. You can already get run over by a forklift if you're standing in front of it. There's various things we do about that, but they're boring real-life things, not fun logic-puzzle things, so they're just not mentioned in the problem. There isn't a way to categorically prevent machines from accidentally killing people though.