I actually don't think it is the direct damage from fighting that gets them but more the inability to stop and scale your commitment to what you can and can no longer do.
Direct damage from combat sports can catch up to you eventually. CTE is pretty serious, but unfortunately you probably don't realize the extent of that damage until it's far too late.