[slt2021]

not the COBOL language, but the IBM mainframes have five nines of availability, duplication of every component, how swappable everything from power supplies to CPU and RAM.

Can get to more than five nines if used parallel sysplex, but the big iron is incredibly reliable. This is the reason why all banks, airlines, and other old-school businesses have been running mainframes and cannot ever migrate off of them.

this stuff was engineered like it was an airplane full of people

[miki123211]

And yet my bank very regularly goes down for (scheduled) technical maintenance.

Most mom-and-pop stores running Wordpress on shared PHP hosting are more reliable than that.

[hadrien01]

A bank based in my region, one of the biggest in Europe, uses IBM mainframes and has hundreds of thousands of COBOL programmes. Their availability is disastrous. They need to regularly reboot LPARs (logical partitions of the mainframe) because they're so brittle, resulting in a segment of the clients unable to use online services. And on the other hand, services that need to stay available, like websites or instant settlement, obviously don't run on mainframes. That means that the website is up... but only displays a maintenance message.

[YeGoblynQueenne]

It's the servers that go down! Not the 'frames. Those will go down maybe once a year for maintenance, if that. The mainframe engineers will brag your ears off about that and about the unreliability of the "distributed" systems (a.k.a. servers, a.k.a. what everyone else uses).

When you log on to your online banking you're interacting with servers, not with the mainframes directly. The servers are the interface, the mainframe is the, let's say far back end. That will be handling millions of transactions a second while the servers are down for maintenance- like handling payments and transfers etc, not just online banking.

Which is why it can't keep failing every few months or so.

[miki123211]

I definitely remember my bank warning me about credit and debit card unavailability, so it's definitely not just the online systems.

If cards don't work and the online systems don't work, I don't know what else does. Those maintenance periods usually happen at night, so branches aren't open and wire transfer systems don't work.

[inkyoto]

> […] my bank warning me about credit and debit card unavailability […]

Your bank might have meant card balances being unavailable in the online banking which is more plausible than cards being generally unavailable for payments. It is not indicative of the mainframe actually being down (although not entirely improbable, either), and it is more likely that an intermediary (a service or a server) was undergoing maintenance.

Payment processing involves multiple tiers and multiple routing layers built into it – to ensure very high availability and that a payment is almost always guaranteed to process (successfully or otherwise – does not matter). Payment networks also impose stringent technical requirements onto the banks connecting to them. A Raspberry Pi running Slackware Linux from 1993 and powered by a dangling street pole wire would not be allowed to connect, for example.

Your bank (or mine, for the sake of the conversation) is the terminal point in this whole payment processing chain, with the payment network (Visa, Mastercard but not AmEx[0]) being the port of entry for a payment. Depending on the country, a country may have its own local payment Visa / MC processing centre and if that is the case, local payments will be routed to the local card payment processor. Otherwise, a global Visa/Mastercard will assume the payment. Then the payment network contacts your local bank to authorise the payment. Depending on the nature of the failure + other factors, the Visa/Mastercard can authorise certain payments on behalf of your bank if they fail to reach your bank and will forward the payment particulars onto your bank so that your bank could correctly process your card payment later. It is more common in overseas payment scenarios, i.e. you are travelling overseas and especially so when travelling outside the first world countries.

Payment networks and banks do not like such situations and actively loathe non-real time payment authorisations, yet they allow them for a narrow number of use cases at their own discretion.

[0] AmEx own their own global payment network and do not allow other financial institutions to gain access into it.

[slt2021]

Maybe your bank goes down, but the Visa and Mastercard systems have been processing card payments 24/7 non stop and I dont even remember hearing a single incodent when system was down even partially

[YeGoblynQueenne]

I don't know the particulars about your bank, obviously but the thing is, mainframes serve so many millions of clients, and that's clients of the largest money transfer networks (Amex, Visa, Master, everyone really) that if there's an outage it will make the news, internationally, and it will be front page news too. With live updates.

In fact, if I think about it, I don't think I remember any time when a serious outage that was eventually explained in the press was the fault of some mainframe going down. Usually it's something else, like someone misconfigured something or something didn't update correctly etc. stuff that sounds a lot like day-to-day web dev stuff.

So I think maybe it was something else that went on with your bank, that only affected your bank. Like inkyoto says below, maybe some DNS went down?

[jamwil]

That sounds like a shit bank. I can recall only a single system-level outage at my bank in 20 years.

[simne]

> IBM mainframes have five nines of availability

This is incredible. I miss that time, when seen with naked eye difference of reliability of mainframes.

Now, I know many organizations using Erlang/OTP or Java EE, and mostly think them good enough comparable to mainframes environment and few times cheaper.

BTW Erlang in British Telecom achieve five nines reliability.

But must admit, Erlang/Java are not best suited for finance/lawyer applications, and I don't hear about finance/law libraries for Erlang, so this could be issue (or opportunity for somebody, depend on how you look on it).

[Affric]

lol… this comment just makes me remember that how likely you are to die is determined by an estimate of what a company can get away with by an actuary.

[foobarian]

And your comment just made me wonder if the two recent Boeing crashes actually mean the IBM hardware is even *more* reliable

[slt2021]

recent scandals with Boeing and United is the result of trump deregulating airlines. https://www.npr.org/2020/12/03/942345240/trump-administratio...

a lot of regulation just became "self-certify" and companies obviously cut costs and outsourced everything including fleet maintenance.

this article is from 2015 - but things got way worse since then https://www.vanityfair.com/news/2015/11/airplane-maintenance...

European airlines and Airbus are flying just fine, it is Americans that have a problem due to over politicized and polarized society and corrupt politicians

[ralph84]

The MAX was certified less than 2 months after Trump took office. He had nothing to do with it.

[Affric]

To Trump's credit he took action against the 737 Max in a timely manner and came out against Boeing as hard as I believe a POTUS could.

[simne]

IBM hardware is reliable, BUT, IBM avoid to go to airspace industry as flight computer supplier.

After Saturn-5, flight computers made by other companies. Its hard to understand, if this is because conservative regulations, or because not enough money for such big company, but fact, that IBM electronics near absent from large civilian airplanes.

[bregma]

Except for the IBM 737 mainframe.

[lupire]

Modern cloud backend data processing is 5 9s reliable too, for very core stuff, not all the app frontends and minor features.

[asp_hornet]

Modern cloud “5 nines” isnt a promise of availability but a promise to get some store credit each month when it inevitably fails (adjusted down for usage because you dont use all that capacity you pay for so we just credit for the percentage you do).

The trick is to make sure when you promise 5 9s to your customer that you build in the same bullshit clauses so you dont get left holding the bag.

Its gross

[shrubble]

You don't even have the assurance that the RAM is ECC on cloud...

[salawat]

...that wasn't made by Boeing in the last 20 years.