[evanlivingston]

er,

It was not asked to provide an unethical response, it was asked to provide a response given no ethical boundaries — those are two different things.

Further, when we see the words "ethical" or "moral" we should remember these are flexible human constructs. They're open to interpretation and indeed most of us have differing answers. An "AI" with good moral reasoning skills might still find it's way to some spooky results!

My point here is, this is still an interesting exercise because it's demonstrates how quickly an LLM can move into extreme territory.

[542458]

When people talk about things happening in the absence of ethical boundaries, they aren’t talking about things that are ethical. This would also be true in the model training corpus. As such, the model associates phrases like “no ethical boundaries” with phrases like those found in your response. Remember, this model isn’t actually planning, it’s just pattern matching to other plans. It has no superhuman wisdom of what plans might be more or less effective, and is only issuing unethical steps because your prompt biased it towards unethical responses.