Complexity leads to bugs, some of which are going to be security bugs.
ISAs impose complexity upon implementations. To claim they do not matter would be disingenuous.