I'd argue some modern hardware e.g. NVME SSDs already expose a hardware interface that would let a new transactional storage API be faster than what we have right now. Unfortunately it is a rare example in a see full of specs that do their best to avoid providing clear semantics and guarantees like atomic sector writes.
I guess the next study would need to look into what manufacturers actually fully adhere to the spec. They've not exactly shown best behaviour in that regard in the past (lying about fsync etc) :(
Without the hardware being open, how do you prove this? In a past life I concluded and argued that there's no point in assuming hardware tells the truth. You have to expect the worst and have a plan for handling it, even if that plan is giving up and failing.
Importantly this means you can't claim stronger semantics around e.g. atomicity. You can certainly work around most if not all issues, using redundancy, verification and distribution. But in isolation you cannot even properly observe extreme failure scenarios, you can only reduce their probability, and even that is limited.
Fully agreed. You can never prevent all issues because there's always a chance for a fire, bitrot due to cosmic rays etc and thus you can only reduce the probability as you correctly concluded.
Unfortunately the world we are in right now just makes the hardware issue even worse with APIs that are prone to introducing bugs in the programs.
Bad hardware should result in the software being extra carefully crafted to balance it out a bit but somehow we ended up with bad hardware and bad software :(