[Doohickey-d]

What are you doing for DB backups? Do you have a replica/standby? Or is it just hourly or something like that?

Because with a single-server setup like this, I'd imagine that hardware (e.g. SSD) failure brings down your app, and in the case of SSD failure, you then have hours or days downtime while you set everything up again.

[kro]

Hetzner normally advertises their hardware servers as 2x 1 TB SSD, because it's strongly recommended to run them in SWraid1 for net 1TB. (Their image installer will default to that)

Once the first SSD fails after some years, and your monitoring catches that, you can either migrate to a new box, find another intermediate solution/replica, or let them hotswap it while the other drive takes on.

Of course though, going to physical servers loses redundency of the cloud, but that's something you need to price in when looking at the savings and deciding your risk model.

And yes, running this without also at least daily snapshotting/backup to remote storage is insane - that applies to cloud aswell, albeit easier to setup there.

[linsomniac]

For over a decade I ran a small scale dedicated and virtual hosting business (hundreds of machines) and the sort of setup you describe works very well. Software RAID across 2 devices, redundant power supplies, backups. We never had a significant data loss event that I recall (significant = beyond user accidentally removing files).

For quite a while we ran single power supplies because they were pretty high quality, but then Supermicro went through a ~6 month period where basically every power supply in machines we got during that time failed within a year, and replacements were hard to come by (because of high demand, because of failures), and we switched to redundant. This was all cost savings trade-offs. When running single power supplies, we had in-rack Auto Transfer Switches, so that the single power supplies could survive A or B side power failure.

But, and this is important, we were monitoring the systems for drive failures and replacing them within 24 hours. Ditto for power supplies. If you don't monitor your hardware for failure, redundancy doesn't mean anything.

[Dylan16807]

> But, and this is important, we were monitoring the systems for drive failures and replacing them within 24 hours. Ditto for power supplies. If you don't monitor your hardware for failure, redundancy doesn't mean anything.

It does still mean something.

If you have a 5% annual chance of failure and no redundancy, your five year failure chance is 23%.

If you have redundancy and literally never check for five years, your five year failure chance is 5%. That's already a huge improvement. If you do an inventory of broken parts twice a year, still no proper monitoring, it goes down to 0.6%

For 2% the numbers are: 10% 1% 0.1%

For 10% the numbers are: 41% 17% 2.6%

(The approximations for small percents are x*5, x²*25, and x²*2.5)

[faangguyindia]

Do not rely on raid alone.

Have 2x servers atleast then invest in proper monitoring.

Server can fail without disk failures.

[kijin]

If that's the tradeoff they're willing to make, who are you to say that they're doing it wrong?

Not every app needs 24/7 availability. The vast majority of websites out there will not suffer any serious consequences from a few hours of downtime (scheduled or otherwise) every now and then. If the cost savings outweigh the risk, it can be a perfectly reasonable business decision.

A more interesting question would be what kind of backup and recovery strategy they have, and which aspects of it (if any) they had to change when they moved to Hetzner.

[hnthrow0287345]

It's possible no one will care much if it's down even for that long. I couldn't care less if my HOA mobile app was down even for a week for example. We don't need constant uptime for everything.

[acdha]

Don’t forget that integrity matters as much as availability in many applications. You might not mind if your HOA takes time to bring a server back up but you’d care a lot more if they lost the financial records or weren’t able to recover from a ransomware attack.

[izacus]

Hetzner provides backups for VPS and machines across all tiers, which are very easy to set up.

[wat10000]

I agree with the overall sentiment, but having an HOA app go down around the time when dues need to be paid could be a serious issue.

[traceroute66]

> Because with a single-server setup like this, I'd imagine that hardware ...

Yeah. This blog post reads like it was written by someone who didn't think things through and just focused on hyper-agressive cost-cutting.

I bet their DigitalOcean vm did live migrations and supported snapshots.

You can get that at Hetzner but only in their cloud product.

You absolutely will not get that in Hetzner bare-metal. If your HD or other component dies, it dies. Hetzner will replace the HD, but its up to you to restore from scratch. Hetzner are very clear about this in multiple places.

[treesknees]

For the price, they could buy an exact replica bare metal server and still save money.

[traceroute66]

> they could

They could, but they didn't and instead they wrote that blog post which, even being generous is still kinda hard to avoid describing as misleading.

I would not have written the post I did if they had presented a multi-node bare-metal cluster or whatever more realistic config.

[locknitpicker]

> They could, but they didn't and instead they wrote that blog post which, even being generous is still kinda hard to avoid describing as misleading.

What do you feel was misleading?

[wiether]

That they get the exact same level of service for $1,199 less per month.

They don't.

And reading the article, they don't seem to understand that.

[traceroute66]

> What do you feel was misleading?

Erm. I already spelt it out in my original post ?

I'm not going to re-write it, the TL;DR is they are making an Apples and Oranges comparison.

Yes they "saved money" but in no way, shape or form are the two comparable.

The polite way to put is is .... they saved as much money as they did because they made very heavy handed "architectural decisions". "Decisions" that they appear to be unaware of having made.

[Someone1234]

They could but then that exchanges cost savings for complexity. You now need to keep them in sync and it is double the cost.

I agree with the other poster, this is fine for a toy site or sites but low quality manual DR isn't good for production.

[daneel_w]

Surely you must've noticed that pretty much all of their bare metal offerings ("dedicated" and the stuff on "auction") have multiple disks, allowing for various RAID configurations?

[traceroute66]

> Surely you must've noticed that pretty much all of their bare metal offerings ("dedicated" and the stuff on "auction") have multiple disks, allowing for various RAID configurations?

I don't know where to start with this comment. Do I really need to spell out the difference between cloud and bare metal ?

A few examples...

    - Live migration ? Cloud only.
    - Snapshots ? Cloud only.
    - Want to increase disk space ?  Tick box in cloud vs. replace disks (or move to different machine) and re-install/restore in bare metal....
    - Want to increase RAM ? Tick box in cloud vs. shutdown, pull out of rack, install new chips (or move to different machine and re-install/restore)....
    - Want to upgrade to a beefier processor ? Tick box in cloud vs move to a completely different machine and re-install/restore

[array_key_first]

You can get snapshots and live migrations working on-prem. The cloud isn't magic, it's just servers with hypervisors and software running on top of them. You can run that same software.

Also, with something like Hetzner you would not be going in and physically doing anything. You also just tick a box for a RAM upgrade, and then migrate over or do active/passive switch.

The cloud does have advantages, mostly in how "easy" it is to do some specific workflows, but per-compute it's at least 10x the cost. Some will argue it's less than that, but they forget to factor in just how slow virtual disks and CPU are. Cloud only makes sense for very small businesses, in which the operational cost of colocation or on-prem hosting is too expensive.

[itsthecourier]

cloud vs bare metal is:

are you a capable engineer or do you believe in magic?

the savings of a cheap engineer disappear on the cloud bill. get a badass well paid engineer who can do both and doesn't talk his way out of this financial madness

[traceroute66]

> get a badass well paid engineer who can do both

Well, fine, but its abundantly clear that this blog post was not written by a "badass well paid engineer".

The person who wrote that blog post was clearly unaware of the trade-offs of the decisions he was making.

[senko]

Well you did say your data is lost when a disk fails, which is not true. Parent pointed out that for you.

Yeah you pay for and get additional stuff with cloud. Nobody disputed that.

[traceroute66]

> Well you did say your data is lost when a disk fails, which is not true.

Well, technically its still a possibility.

I am old enough to have seen issues with RAID1 setups not being able to restore redundancy, as well as RAID controller failures and software RAID failures.

Also, frankly you are being somewhat pedantic. My broader point was regarding cloud. I gave HD Failure as one example, randomly selected by my brain ... I could have equally randomly chosen any of the other items ... but this time, my brain chose HD.

[faangguyindia]

You can just run 3 dedicated servers and design your app so that it never fails.

[andai]

Can you elaborate? I'm coming up with similar designs recently (static site plus redundant servers) but my designs so far assume no database and ephemeral interactions. (Realtime multiplayer arcade games.)

Curious what the delta to pain-in-ass would be if I want to deal with storing data. (And not just backups / migrations, but also GDPR, age verification etc.)

[faangguyindia]

database isn't hard to have HA with, it's actually very easy to do any of this.

i already design with Auto Scale Group in mind, we run it in spot instance which tend to be much cheaper. Spot instances can be reclaimed anytime, so you need to keep this is kind.

I also have data blobs which are memory maped files, which are swapped with no downtime by pulling manifest from GCS bucket each hour, and swapping out the mmaped data.

i use replicas, with automatic voting based failover.

I've used mongo with replication and automative failover for a decade in production with no downtime, no data lost.

Recently, got into postgres, so far so good. Before that i always used RDS or other managed solution like Datastore, but they cost soo much compared to running your own stuff.

Healthchecks start new server in no time, even if my Hertzner server goes out or if whole Hertzer goes out, my system will launch digital ocean nodes which will start soaking up all requests.

[faangguyindia]

The easiest I’ve done is in MongoDB replication, sharding, failover, and all that is super easy.

Recently, I did it in PostgreSQL using pg_auto_failover. I have 1 monitor node, 1 primary, and 1 replica.

Surprisingly, once you get the hang of PostgreSQL configuration and its gotchas, it’s also very easy to replicate.

I’m guessing MySQL is even easier than PostgreSQL for this.

I also achieved zero downtime migration.

[acdha]

Replication is not a backup. It helps for migrations or clean single node failures but not human error, corruption, or an attack.