[tbrownaw]

> distribute batch workloads in parallel across a cluster of machines - no other software and OS did such a thing, maybe still doesn't

What am I missing?

One of my college courses ages ago was about working with MPI which we got to run on the hpc cluster.

The last time I needed to run N copies of something I asked kubernetes to do it. The time before that, I asked $cloud_vendor for N identical VMs with the same cloud-init script.

Supposedly Google's in-house stuff that kubernetes and map-reduce (the product, not the concept) are public versions of, is all about running stuff well on huge groups of machines.

[anyfoo]

I can't speak for OP or their particular use case (because I don't know about it), but kubernetes is not a panacea for large scale batch processing.

For example, large banks need to securely, reliably, and very efficiently process an unfathomable amount of transactions[1]. In this case, kubernetes would be a giant waste of resources and complexity. The former one hampers throughput, the latter one means security and reliability suffer.

For people not familiar with it (me included, actually), it can be mind-boggling what throughput is achieved, and what mechanisms for reliability are in place. Not just in software, in actual hardware; this goes way beyond ECC memory.

[1] Transactions in the bank sense, not in the computer sense, because I don't want to confuse matters more. In the mainframe world for example, there is a difference between "batch processing" and "online transaction processing", but both could be applied to bank transactions. Note that I'm not advocating for the mainframe world here.

[anonymousDan]

Interesting, can you give rough order of magnitude for the txn/s throughout achieved? Would also be really interested on more info or pointers to the hardware reliability mechanisms!

[stcroixx]

MPP was the acronym for the pattern. It let us take a binary executable which would normally just be a single PID on one node and it would distribute the PID and all its resources including files across however many nodes you had configured. The software that managed this was heavily dependent on RISC processor architecture which basically meant only AIX RS6000. Googles solution is designed for commodity processors. The modern equivalents let you do something sort of similar, but you have to design for that and as far as I know you’re not sharing resources like open file handles across the cluster. We took C programs and ran them as is.