TrustZone is essentially an ISA extension, similar to Intel's TXT and SGX to provide a trusted execution environment. You can trivially avoid it by never running any of the related instructions.
It's more of an extra address bit and processor mode. It doesn't have related instructions like TXT and SGX, but instead is structured more like a hypervisor.
OK, so there's the one instruction to do a system call that hits secure mode. It's equivalent to svc or hvc, but hits EL3 (secure mode) rather than EL2 (hypervisor mode) or EL1 (supervisor mode).
It's very very different than the dozen or so instructions to setup TXT or SGX that sits off to the side of the main OS rather than running like a super hypervisor. If you're going to compare it to something, it's way more like SMM on x86.
Source: I've ported a kernel to EL3 (secure mode).