No, the Xeon Phi "accelerators" are usable too, they are basically 486 cores on modern litography (to allow for higher density/clock speeds), with a vector unit attached to them. I don't know how hard it would be to boot linux on one though...
This host system should be not much more than a PCIe root emulator though. This is the level one can get on an e.g. FPGA with custom logic, which implies that any attempts to insert hardware/firmware level attacks into the actual logic you care about is near impossible to do due to the low-level nature of the custom PCIe implementation.