It's also easy to do this with Tarsnap. You can split out your host keys with tarsnap-keymgmt into separate read, write, and delete keys. Just take your delete key off of the host and store it somewhere secure.
Or better yet, create the key somewhere else and only upload the keys you want to have on the server. Otherwise there's a chance that the original tarsnap key is left in an unallocated block.