[skissane]

Yes, they do use Fibre Channel (FC), but the protocol they run on top of FC is very different from the usual FC protocols. FC has four layers, from FC-0 (the physical layer) through to FC-4 (the application layer). At level FC-4, Linux/Unix/Windows/etc servers mostly speak "Fibre Channel Protocol for SCSI" aka SCSI-FCP, which transmits SCSI commands over FC. However, while IBM mainframes do use SCSI-over-FC, they also use something else called "FICON", which instead of transmitting SCSI commands, uses the same command set as IBM's legacy non-SCSI mainframe hard drives (such as the IBM 3390, which was IBM's last non-SAN mainframe hard disk line from the late 80s / early 90s).

Mainstream platforms have what IBM calls FBA (Fixed Block Array) disks, in which the disk is an array of sectors which all have the same size. While the IBM mainframe hardware supports that, most IBM mainframe operating systems don't support those, however; Linux and VSEn are the exceptions, and z/VM is a partial exception.

As well as FBA, IBM has ECKD (Extended Count Key Data) disks, like the IBM 3390 was. (ECKD was preceded by CKD, which is conceptually the same – the same sector format, etc – but had a less efficient command set.) With ECKD, sectors can have variable sizes – each disk track can have different sized sectors, you can even mix different sector sizes within the same disk track. Also, sectors optionally have keys, and the disk has commands to search for sectors based on their keys. Originally, these variable sized sectors actually physically existed on disk; nowadays, the SAN only has standard hard disks (or SSDs), but ECKD disks are emulated in the SAN software on top of them. z/OS, the most popular IBM mainframe OS, only supports ECKD disks, not FBA ones. Linux and VSEn support both ECKD and FBA. For ECKD disks, you can't use the standard SCSI command set, you have to use IBM's DASD command set.

Many high-end enterprise arrays do support ECKD and FICON, but you usually have to pay significant extra license fees to enable the software that supports that.

[HyperSane]

Hearing people talk about IBM mainframes makes me realize what I sound like talking about my job to my mom. IBM could have 90% of the home computer and server market if they didn't charge so much for their stuff. They always had the best technology.

[skissane]

> IBM could have 90% of the home computer and server market if they didn't charge so much for their stuff. They always had the best technology.

In this particular case, there is no contemporary technological advantage of ECKD over standard hard disks/SSDs - it is simply a matter of legacy backward compatibility, lock-in, and an attempt to financially sustain the mainframe storage ecosystem. The idea of variable sector sizes may have had some real advantages when it was invented back in the 1960s, but nowadays it is just adding unnecessary complexity for no real benefit.

Even the idea of having the hard disk do key searches is something IBM has been moving away from, because doing them in software on the CPU turned out to be much faster in practice. It is still required because the primary traditional mainframe filesystem (VTOC) is based on it, but in areas where performance is critical (such as databases) IBM now positions it as a legacy technology.

Maybe there's still some value in moving key searches into the storage layer, but if there is, it would have to be something much more advanced than the rather simplistic 1960s implementation that ECKD storage provides. Oracle came up with a conceptually similar idea much more recently, Oracle Exadata (released 2008), in which some aspects of database query execution are delegated to the storage–especially table scans. ECKD only supports very basic comparison operations on the key field; Exadata, from what I understand (I used to work for Oracle, but not in this area) can evaluate complex predicates over the whole database row.

[HyperSane]

I worked for a state government that was in the process of migrating hundreds of Oracle DBs from VMWare VMs to Exadata servers. I think combined they had over 20TBs of RAM. I asked one of the DBAs about how much better performance was and he said he typically saw at least a 10x increase in QPS and reduction in latency.

Samsung has a SmartSSD with a FPGA on it to offload compute to the drives.

https://phisonblog.com/why-is-computational-storage-inevitab...