Physical access means you can directly measure things that would otherwise be secret. Nothing stops you from reading the contents of memory wholesale, for one simple example.
For some things we are, but I don't know how far along open source software is currently. It would take a major incentive to get people to even talk openly about what's possible in this space.
CPU Technology like Intel Software Guard Extension or AMDs Secure Memory Encryption closes that gap as well. It's not magic and might be flawed in its current forms - I'm no expert on that - but I do believe in a not too far away future physical access will no longer provide cheap attack vectors.