[Terr_]

I think the key is that models can be forced to reveal arbitrary information, even if the information they have is--for now--mostly public information anyway.

For contrast, imagine an LLM model trained on every top secret document ever. It's important to know if "don't reveal information the user isn't allowed to see" is a crazy impossible dream of so-called prompt engineering.