[conformal]

i strongly suggest that sites concerned with their security use (1) full-on colocation services and (2) encrypt the partition they store user data on or encrypt the entire disk.

if you have proper disk encryption, it is non-trivial to game the remote physical access to the machine, e.g. attacker convinces someone to use 'remote hands' to reboot server and then gets console redirected pre-boot. if you have disk encryption in this scenario, whether it is on the user data partition or the whole disk, you will surely be notified of the unscheduled reboot and can investigate it.

it is always best to host your own machines (_not_ VPSes) and be able to provide some level of compartmentalization to your hosting setup.

[stefanha]

Careful, disk encryption usually doesn't cover the entire disk. So an attacker can place an evil initramfs in the /boot partition that stores away your disk encryption passphrase, for example.

This is even easier if the operating system partition is plaintext and only the data partition is encrypted. Then it's trivial to modify any binary, library, or startup scripts!

Encryption just means an attacker cannot get at the data right away. But once the admin brings up the system again (not knowing something has been tampered with) it's pretty easy to get access.

[hbbio]

This is exactly the reason why they say they are not restarting the site now but are building a new fresh server instead.

[rdl]

Of course, there are availability concerns with that, which you can mitigate with onsite redundancy (dual power supply machines fed separately, etc.), and multi-site replication.

The big issue is how to get keys back into the system. I suggest having some onsite tamper-resistant component and some typed in remotely component.

The annoying thing is there is no viable low-end solution for this yet. It makes sense once you're dropping $100-150k of equipment per site, but for 2 servers per site, it's a huge pain and lots of overhead.

[conformal]

i always advocate for FDE, but that often has issues with remote serial console. the threat model of running without disk encryption is far worse for most bitcoin-related sites than the complexity associated with redundancy. if they get hacked, they are likely going to eat downtime anyhow.

as far as low-end solutions are concerned, a usb serial console adapter plus a few machines runs about USD 5K. set machines to redirect console to serial and have an OOB machine for unlocking downed servers.

i'd be interested to hear what kind of solutions there are for onsite tamper-resistant components.