[skissane]

If I had more money than sense, I’d do the same thing.

If I was filthy rich (which I’m not and probably never will be), I’d build a data centre in my mansion and buy a brand new IBM mainframe and stick it in there. Because I can. For the same reason other rich people build some huge garage with dozens of cars in it.

[HyperSane]

I would buy one of every high end enterprise storage array to play with. My former life as a VMWare admin made me appreciate storage.

[skissane]

Which means you'd need a IBM mainframe, since IBM mainframes require enterprise storage arrays which come with special software for emulating the old mainframe hard disks (ECKD), and if you are going to buy one of those, you'd need a mainframe to drive it.

[HyperSane]

No, most of them use fiber channel.

[skissane]

Yes, they do use Fibre Channel (FC), but the protocol they run on top of FC is very different from the usual FC protocols. FC has four layers, from FC-0 (the physical layer) through to FC-4 (the application layer). At level FC-4, Linux/Unix/Windows/etc servers mostly speak "Fibre Channel Protocol for SCSI" aka SCSI-FCP, which transmits SCSI commands over FC. However, while IBM mainframes do use SCSI-over-FC, they also use something else called "FICON", which instead of transmitting SCSI commands, uses the same command set as IBM's legacy non-SCSI mainframe hard drives (such as the IBM 3390, which was IBM's last non-SAN mainframe hard disk line from the late 80s / early 90s).

Mainstream platforms have what IBM calls FBA (Fixed Block Array) disks, in which the disk is an array of sectors which all have the same size. While the IBM mainframe hardware supports that, most IBM mainframe operating systems don't support those, however; Linux and VSEn are the exceptions, and z/VM is a partial exception.

As well as FBA, IBM has ECKD (Extended Count Key Data) disks, like the IBM 3390 was. (ECKD was preceded by CKD, which is conceptually the same – the same sector format, etc – but had a less efficient command set.) With ECKD, sectors can have variable sizes – each disk track can have different sized sectors, you can even mix different sector sizes within the same disk track. Also, sectors optionally have keys, and the disk has commands to search for sectors based on their keys. Originally, these variable sized sectors actually physically existed on disk; nowadays, the SAN only has standard hard disks (or SSDs), but ECKD disks are emulated in the SAN software on top of them. z/OS, the most popular IBM mainframe OS, only supports ECKD disks, not FBA ones. Linux and VSEn support both ECKD and FBA. For ECKD disks, you can't use the standard SCSI command set, you have to use IBM's DASD command set.

Many high-end enterprise arrays do support ECKD and FICON, but you usually have to pay significant extra license fees to enable the software that supports that.

[HyperSane]

Hearing people talk about IBM mainframes makes me realize what I sound like talking about my job to my mom. IBM could have 90% of the home computer and server market if they didn't charge so much for their stuff. They always had the best technology.

[skissane]

> IBM could have 90% of the home computer and server market if they didn't charge so much for their stuff. They always had the best technology.

In this particular case, there is no contemporary technological advantage of ECKD over standard hard disks/SSDs - it is simply a matter of legacy backward compatibility, lock-in, and an attempt to financially sustain the mainframe storage ecosystem. The idea of variable sector sizes may have had some real advantages when it was invented back in the 1960s, but nowadays it is just adding unnecessary complexity for no real benefit.

Even the idea of having the hard disk do key searches is something IBM has been moving away from, because doing them in software on the CPU turned out to be much faster in practice. It is still required because the primary traditional mainframe filesystem (VTOC) is based on it, but in areas where performance is critical (such as databases) IBM now positions it as a legacy technology.

Maybe there's still some value in moving key searches into the storage layer, but if there is, it would have to be something much more advanced than the rather simplistic 1960s implementation that ECKD storage provides. Oracle came up with a conceptually similar idea much more recently, Oracle Exadata (released 2008), in which some aspects of database query execution are delegated to the storage–especially table scans. ECKD only supports very basic comparison operations on the key field; Exadata, from what I understand (I used to work for Oracle, but not in this area) can evaluate complex predicates over the whole database row.

[remram]

It sounds like they wanted 2 separate network paths but they still end up with only 1 where they can use BGP. Maybe they can still use their residential connection as failover, with a VPN to a VPS where they can use BGP?

[dboreham]

They all likey traverse the same physical path and would all be taken out by the same gravel truck hitting a pole. The two connection thing was I think more about defense against random retail connectivity flaps (someone decides to upgrade a router's firmware and just takes the town offline, meanwhile nobody answers support calls, that sort of thing).

[bombcar]

Yeah true multihoming is expensive and involves things like hiring people to actually inspect the physical lines and Make sure they don’t run down the same street at some point (and be subject to the same backhoe of fiber death)

[remram]

True. I thought they wanted BGP to implement multihoming but upon re-reading they needed it anyway for other reasons.

[dboreham]

BGP would be to allow fail over of some service from their primary colo.