[Beretta_Vexee]

I think it's better to think of it as a fibre channel protocol rather than TCP. It's intended for use on managed internal data centre networks. It skips OSI layers to gain speed and probably do 100% hardware routing with FPGAs.

It's of no interest on the internet or any small scale netwwork.

[KaiserPro]

> fibre channel protocol

Apart from FC is is explicitly lossless and ordered

[bcrl]

FC is not entirely lossless. One ticket I had the joy of dealing with involved a customer using a Fibre Channel network for their storage using multipathd for failover. In theory it was a fully redundant configuration with dual FC ports on the server with each one going to a different FC switch all the way back to the SAN. However, the system was generating I/O errors on large writes while small writes would succeed. Needless to say that ext4 failed horribly, and there were worries that it was a kernel bug in the FC driver.

After a good amount of back and forth with the customer, and several test programs run on the system in question, I eventually came up with a hypothesis that there was an error in the write path of the SAN as small writes succeeded while larger writes failed. The customer ultimately found there was a dirty fibre on one of the links in their FC fabric. It was dirty enough to corrupt large packets, but not so dirty that smaller writes and control packets were unable to get through. Since multipathd only checks to see if a given target can be read from, it would never fail over to the other path (which was fine). So much for trying to build a high availability system using an expensive SAN!

Lesson of the story: what you think is a lossless network is not always lossless. Using the IP stack has a lot of beneficial diagnostic tools that you really start missing when something goes awry in a non-IP network.

[stonogo]

Broken hardware does not make the protocol lossy. I think you're misunderstanding what 'lossless' is intended to mean in this context; it does not mean that it is error-free. In a lossy protocol, missing data is not necessarily an error. In a lossless protocol, missing data is treated as an error, which is consistent with what you experienced.

[bcrl]

I do understand what lossless means. The point of my anecdote is a tale of warning that when going off and start designing new network protocols, especially one as bare bones as TTPoE, you need to consider what happens when someone has to deal with things going wrong. Diagnostics and maintenance matter in the real world for people running large systems with thousands or millions of moving parts. IPv4 and IPv6 bring along lots of tools that help in these scenarios, and IPv4/v6 headers don't actually have all that much overhead to parse and generate in hardware, plus they are protocols that have been around long enough to have many widely available hardware and software implementations in open source or to be purchased from vendors. I'm certain that there will be times when sysadmins will be cursing the fact that the folks who implemented TTPoE didn't have a ping-like tool available from the start.

[stonogo]

FC remains a lossless protocol; bugs in multipathd just mean we live in an imperfect world. Your initial sentence, "FC is not entirely lossless," conflates a specific networking term of art with a pedantic application of the denotative definition of the word. If your point was that immature network technologies do not have as many diagnostic tools as mature ones do, you should have made that point instead of misappropriating jargon.

Anyway, to your specific point, IP at all is basically overkill in a cluster architecture. Very few IP stacks function properly without having to get things like ARP involved; the more of this stack you can get rid of, the better performance you get and there's less to maintain. TTPoE reminds me the most of ATA over Ethernet, a previous effort to shed the complexity of a protocol designed for global networking. It worked great until you hit scaling issues, which competing tech leveraged the aforementioned complexity to address.

[bcrl]

At a high level, FC lost the write without responding to the write in a timely fashion while other I/Os went through. The write request never gets through from the host to the target, and the host ends up timing out the write then throws an I/O error => that seems like a lost write to me at a higher level. Lossless only applies at the lowest layer of the stack; any holistic view of the system would view this scenario as lossy.

I have implemented ARP and UDP on FPGAs for some toy projects, and it's really not that difficult. One of the use-cases I played around with was getting debug data out of an FPGA at multigigabit rates -- things like PCIe TLPs and raw SERDES data from an EPON implementation to debug a burst mode CDR. The fact that the protocol was IPv4/UDP was no impediment to having it push data through at line rate. Once you've implemented parallel CRC32 for ethernet packets from scratch on a 256-512 bit wide data bus where packets can start and end on arbitrary 32 bit boundaries, the complexity of IPv4 and UDP checksums is dead simple in comparison.

I understand and agree with throwing out TCP in TTPoE. I do not agree with throwing out IPv4 / IPv6. Heck, you don't even need ARP for v6, you could get away with link local addresses using the ethernet MAC address you already need to have anyways.

[KaiserPro]

FC should be able to detect errors. I've had alerts shout at me when an FC switch detects a dropped packet.

More over, the multi-path should have stopped that! it should have detected a bad link and failed over to the other one (but the config for that is hard, so I can see why that might not worked. )

[bcrl]

Last time I checked, multipathd does not and cannot detect faults on the write path as it only performs small reads to check the health of any given path. Checking writes would involve allocating space on the disk for multipathd to safely write to. Maybe someone has changed that in the past decade? I don't know as I'm not involved in anything SAN related anymore (and thank goodness for that!). SAN hardware is particularly awful as the underlying network is essentially hidden from the operating system most of the time. Storage subsystems built 30 years ago were built without any consideration that they might running on top of networks.

These and many other performance issues left me with a particular hatred of SANs.

[KaiserPro]

yeahnah, I totally feel your pain.

[delfinom]

Elon just doesn't want to pay Nvidia for Infiniband. Lol

[andix]

If it works and it's cheaper, this is a very reasonable thing to do.