[graypegg]

Yeah, though that surprising effect is present in any LLM. You can give it text, and it gives you more text that's spookyly coherant and related. Some of it is going to be in first person because lots of it's training data was in first person. That's still neat! But...

The thing Anthropic really wants us to believe is that rather than just feeding the text it outputs back in, which is a rather banal framing of what they're doing, we've "given it a secret notepad". It's a narrative framing that I think obscures the REAL interesting stuff going on, but I guess LLMs are now too boring now so we need to create some pointless moral drama about it for press.