That's the problem : you would have to trust your assembler. How was the assembler compiled or assembled ? Etc etc. You would go back to the first "physical" translation of a program "into" a computer.
You'd have to write an assembly program and then a hand-translated binary version that can run directly on the bare metal with no OS. And use it to compile the "real" compiler.
I wonder if one could make that simple enough to do be "somewhat reasonable," yet complex enough to compile the real compiler. Probably! Though it may take a team of people quite some time.
I wonder if government(s) have infrastructure/teams that do this already, or if there are any open source projects aimed at this kind of thing.
1. Trusting chip fabs is a non-starter. Institutional security only helps large institutions. You will not audit your fab, nor can you trust other people to do it for you. Unlike software, there is no "certificate" that is easily verified in the event of a backdoored fab.
2. There's two phases in the lifecycle of a backdoor. First, is its latent 'offline' period which is noninteractive with respect to the attacker. For example, a backdoored compiler [RoTT] basically propagates a virus to new copies of the compiler. Solving this seems tractable by eliminating bootstrapping as the sole method for compiling a compiler, and research into interpreted languages that are easily ported to new platforms for the stage0 compiler.
3. The second phase is when the introduced vulnerability is 'active' and ready to be exploited over the network by an interactive attacker. The case of a network-facing compiler-infected binary should be easily solved by solving the first problem. The case of backdoored hardware/microcode is much more insidious, and requires coming up with assumptions to frame the problem for even a chance at being tractable. (Also, it requires the trustable software tools from (2) for implementing the solution)
4. Secure 'offline' computing on its own would be a boon for things like maintaining the integrity of master cryptographic keys (although keep in mind one still has to prevent against a backdoored CPU acting as an infected stage0, but this is much easier in a noninteractive setting). I'm less familiar with state of the art in this area than I should be, but I'm guessing implementations are probably still in the dark ages of 'trust the company'.
I think that it boils down to writing arbitrary data in memory (so that they can't follow the usual path of programs, as with compilers). For the anecdote, a friend of mine used to flash a "hello world" program into a microcontroller using push buttons only :)