[somat]

I think the biggest problem is the various filesystem layers assume the system will be more reliable than it is and suffer. I have a fun weekend experiment running, encrypted remote block storage where I put softraid crypto device(yes I like obsd) on a iscsi target from across the internet. I was vaguely surprised it works at all. But it tends to lock up often as something in the raid layer fails to understand iscsi hiccups.

A fun thought experiment here. Is an encrypted transport layer needed? (ipsec, wiregaurd, tls, etc...) I mean, sure, it probably should be plumbed in I bet a raw iscsi stream exposes a ton of metadata. But does it need to? Would a system designed to encrypt blocks in storage provide protection for those blocks in transit?

[qdotme]

Well, sort of. I think a big portion about it for me was figuring out the TaskAttribute and ACA handling. Basically, many of these assumptions are (well?!) communicated to the block layer by the SCSI protocol (not even iSCSI, that is a thin layer) - but configuring it on the target side was always poorly understood.

We run "Ordered" - without queueing - so we essentially are trading off some /more/ performance for reliability in this situation. The block layer tends to handle it well in most OS.. I'll be game to test what softraid crypto would do to it (luks actually handles it reasonably!)

In terms of encrypted transport - the value is mostly in HMAC layer. That might die oddly if the ISPs decide to mess around with these packets because they look too cute - but for encrypted FS, it shouldn't be accidental, only malicious. (Accidentally, I've seen ISPs get upset over MPEG streams over netcat!)