Hacker News new | ask | show | jobs
by magicalhippo 872 days ago
"What if I have 3 raspberry pi 4's [...] with no significant performance requirement?"

Ceph is nice, but performance is lackluster on anything but a proper cluster (pun intended).

It's also somewhat heavyweight. I ran ZFS over iSCSI with four RPis serving the iSCSI targets via SATA-USB. It was network limited mostly. The advantage of that is that you can take the same disks and plug them all into a single host and import the pool directly (ie not via iSCSI), if needed.
1 comments
redundantly 872 days ago
I would love to learn more about that setup. Do you have it documented anywhere?
link
magicalhippo 872 days ago
Unfortunately I have not.

The core concept is rather simple though: iSCSI allows for an iSCSI server to expose raw block devices (called targets[1]), so do that and use them from a client machine as-if they were locally connected disks.

So I used LIO[2] as the iSCSI server on each of the Pi's, making sure to use the "by-id" to reference the block device so it would work fine across reboots (skipped changing "directories"):

    sudo targetcli
    /backstores/block> create name=block0 dev=/dev/disk/by-id/usb-SAMSUNG_MZ7PD256HAFV-000_0123456789000000005-0:0
    /iscsi> create
    /iscsi> cd iqn.2003-01.org.linux-iscsi.rpi4.armv7l:sn.fc1b1c9879a1/
    /iscsi/iqn.20....fc1b1c9879a1> cd tpg1/luns
    /iscsi/iqn.20...9a1/tpg1/luns> create /backstores/iblock/block0
Note you might have to use /block/ instead of /iblock/ in the last command there[3], depending on targetcli version.

Then you create the portal which exposes the target (backed by the raw disk) to the network as normal (see Debian guide fex). I did it like this, you have to adjust for the name of your device:

    /iscsi/iqn.20...1c9879a1/tpg1> cd portals
    /iscsi/iqn.20.../tpg1/portals> delete 0.0.0.0 3260
    /iscsi/iqn.20.../tpg1/portals> create 10.1.1.101
    /iscsi/iqn.20.../tpg1/portals> cd ..
    /iscsi/iqn.20...1c9879a1/tpg1> set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1
This creates a portal which exposes a single LUN, backed by the raw disk, on the IP 10.1.1.101, and disables authentication and demo mode (write protection).

Next you install the iSCSI client (called initiator)[4], I did this on a separate machine. There I connected to each of the target on each Pi. Once successful you should get a number of disks under /dev/disk/by-path/ip-*, which you can then specify when importing the pool on the iSCSI client machine, say

    zpool -d /dev/disk/by-path/ip-X -d /dev/disk/by-path/ip-Y
Since you're exposing the raw block device, there's no difference in that regard to having the disk plugged into the client machine directly.

Btw, I recommend not messing with authentication at first, though it's not terribly difficult to set up.

edit: As mentioned the nice thing about this approach is that it's effectively "harmless". If it doesn't work out for you, you can always put all the disks in a single machine and import the pool as normal.

Also, iSCSI has a lot of robustness built in. For example the iSCSI client will temporarily store writes and re-issue once the target (server) is back online. I safely rebooted one of the Pis while copying data to the pool, for example.

[1]: https://en.wikipedia.org/wiki/ISCSI#Concepts

[2]: https://wiki.debian.org/SAN/iSCSI/LIO

[3]: https://github.com/ClusterLabs/resource-agents/pull/1373

[4]: https://wiki.debian.org/SAN/iSCSI/open-iscsi

link