[gwern]

No. The tanks problem is when you are picking up a genuine signal in the data. If you collected more data exactly the same way, then it would continue to predict well on this new data; and many other model architectures would also pick up the signal (because it's genuine).

This is the obvious explanation for their initial results, and one of the first things I said when I saw the preliminary results: "how do you know there isn't some subtle association between 'eagle' and $arbitrary_small_integer that you are just ignorant of but the superhumanly knowledgeable LLMs have learned about?"

But then the later experiments rule that out, in part by showing that it doesn't transfer across different models (ie. initializations).