Yes, rectangle would also be degenerate. Basically any configuration where adding an epsilon weight somewhere generates a non-epsilon shift so to speak.
To me, "degenerate" means that it has a symmetry so that the different states are somehow identical (e.g. equal energy in a quantum system). If it's rectangle-ish (say, an imperfect hand-drawn rectangle) then it has 4 distinct equilibrium positions. Not sure if we're thinking of the same with "degenerate".