What is more risky to you: Leaving known vulnerabilities such as spectre unpatched or the possibility of Intel adding a backdoor for some unknown purpose that wasn't present in the shipped hardware?
The former is more risky from the security point of view. The latter is more risky from the freedom point of view. (And, while an FSF supporter, I choose to be more secure.)
Vulnerabilities such as spectre are only relevant if you run untrusted non-free software. Also these vulnerabilities show that sandboxing is not effective on current CPUs, and specific mitigations does not solve the problem in general.