[ratel]

To be sure I'm not defending the practice, just shedding another light on it.

In answer to your first question. There might not be an insurance against the system failing. The honest answer to what to do then, because someone always needs to ask is: Panic! It might not even be unrecoverable, just really expensive and time-consuming to do so.

The follow-up question would be: Yes it is a risk, yes it is a possible disaster. What do you want to do about it? If your answer is anything more than: We should not have taken on that risk in the first place or lets bet the companies future on the fact that we can fix this. Then it might actually be interesting to listen to. It seems from your comment you resolved it.

As for messaging. I do not agree you should explain the situation on the server. Just let people know they should not touch this ever. As soon as you explain why people will assume their reason to do so will trump whatever reason you gave. If they need to think of all possible disasters that might happen it has more impact than the one you can describe.

Messages like "Before doing anything on this server contact Bob" will eventually lead to Bob receiving a message: "We have done this or that. Just letting you know, but it was after office hours" which he will probably see rushing into the office in the middle of the night because the server is not working anymore. The other type of message: "Don't reboot this server ever! For more information contact Bob" Bob changed because he spend a considerable time saying no and explaining the situation to the sysops team, their manager, their manager's manager, etc. who all thought their priority must trump Bob's. Bob might still be working for the company. He might have tried the better part of his career to get this stain resolved. Nothing bad about Bob.