If your memory doesn't work, what does it matter how much you have?
You can create ungrounded abstractions all you you want, but as an older person who has commiserated with other older people, the common conception that it's harder to learn and retain things in older age is a real thing
Sure, but what you're describing is the difficulty of learning as age sets in. That's very different from the original assertion- "If I start specializing elsewhere I'll forget the intricate details of the thing I'm already specialized in." - unless that person was describing themselves as older.