Other than batch jobs, I can't think of a problem that can be solved these days that doesn't also require high availability - at the very least they require a warm standby.
I dunno, it often matters a lot less than you think when something goes down. And distributed systems have a knack for going down in a much less predictable way, it's not going to automatically make your system more reliable.
(modern server hardware and operating systems are also surprisingly reliable nowadays, which makes it harder to reach breakeven with a distributed design)
I live in the ERP-adjacent world, and the vast majority don't have any of that, never have, and I pray most don't.
P.D: the business world is the one that needs AND tries everything, and most attempt to implement something is much worse than the simpler previous thing.
For most that I know and most I bet (even without knowing) add "high availability" or "warm standby" WILL CERTAINLY lead to worse availability.
In fact, the best setup for most people, and consider things from today (that you will see, not that different from mainframe days!):
- Single server on a *nix LTS/immutable distro with just the RDBMS + App backend, on decent hardware. Fast, lots of ram or whatever optional
- A true reliable backup system
- A decent network setup, ideally a VPN one for connect everybody, with good firewall, SSL and that is
More complex than this and is 90% certain is much worse, other 8% until the dude that do it and care leaves.
As always it depends but at least for me in my anecdotal experience as a Distributed systems engineer (I know, I'm clearly very biased here) - the problem with a single system is that mundane things cause unavailability like - software updates, hardware upgrades/failures, having a single fault domain (IE, someone writes a bad query that wedges the DB, now no one can use it) and since failure isn't built into the design, _when_ things fail the mean time to recovery tends to be rather high. For example, if the power supply explodes and takes out your server, how long would it take to procure a new server and restore from backup? On the flip side, if your system is designed around (for example) server less functions or spot instances- which become unavailable multiple times per day, you would have already engineered in fast recovery
Well if the company have somebody like you it helps tons!
In the flip side:
> the problem with a single system is that mundane things cause unavailability
Is multiplied by each "system" you add. The basic failures are relative easy to deal and understand, the ones introduced by more complex system who knows?
Probably the most important step that I miss:
> How long would it take to procure a new server and restore from backup?
.. and applied to any complex extra you have.
For single system is viable (with something like nix) to go off for maybe half hour for what I see around, most of the time in procuring another machine (that what people do is to Bring any other machine it can not go to amazon and buy!) and restoring the backup.
Of course, I factoring that downtime is not "seconds or minutes" here, but neither I think many can do like that
Was still the take away. The idea is the lost a single instance should cause downtown so it’s not multiplied. IE three instances of the database case tolerate a loss of a single instance for maintenance since the other two will take over the load.
There is of course an argument against the cost of this and sure I’d even accept the complexity argument since you probably need to add more tooling to manage the hand over but again the point of this complexity is specifically to avoid having a single point of failure / to naturally handle failure such that the possibility of downtime isn’t multiplied
So take 16 independent computers and have them each solve the problem separately. You'd still be saving 80% compared to the paper's benchmarks. It wasn't close.
McSherry does a lot of interesting work on making monotonic/incremental distributed systems efficient (e.g. Differential and Timely Dataflow). Those kinds of systems scale much more linearly.
(modern server hardware and operating systems are also surprisingly reliable nowadays, which makes it harder to reach breakeven with a distributed design)