Their actions would most likely be visible on the network because it is transparent. The community can then decide whether to activate a fork and punish the colluding nodes.
I still don't understand how consensus works off the chain. If 51% of the stackers just decide to run their own software and do their own thing. How does something off chain change that? How does something off chain penalize a majority of staked ETH?
If the 51% is a single staker or small group of colluding stakers attacking the chain, the rest of the users will probably not want to continue using that chain. The users can activate a soft fork, just by changing the rules of the code to burn the attacker’s funds, and running the updated client software. The remaining honest stakers can then continue to follow the head of the non-attacked chain, and the attacker would be the only one left on their chain.