Apparently this was just a function of how it was prompted, they claim it works fine if you use a different prompt -- https://www.anthropic.com/index/claude-2-1-prompting