[codegeek]

"I learned my lesson about not doing an offsite backup regularly"

Heroku is a shitshow after the Salesforce takeover and not to shit on you because I know it really sucks. BUT please everyone, do offsite backups and test them. Please people. Please. If you have anything that is important, BACK THEM UP on your own outside of the provider.

Heck, we wrote our own script to backup RDS databases offsite as well even though RDS has backups and restore options. I want that database file.

[justin_oaks]

I spoke out loudly at a previous employer how it's really dumb to have our database and it's backups under a single AWS account. A single AWS account compromise, account issue (e.g. AWS shuts down account), or a disgruntled employee could result in the business being destroyed.

They took some half-hearted efforts to back up the data, but it was far from ideal.

Back up your data, and do it in different places so a failure in one won't affect any other copy of the data.

[robryan]

Yeah pretty cheap and easy, at least relative to the value of data to most companies to setup a server you have physical control of to go download the backup once per day.

[hot_gril]

It's not exactly cheap to always have a backup that you know works. You have to set it up and test it periodically. Of course, that doesn't mean you shouldn't do it.

[fishnchips]

Very much that. As a former SRE responsible for tape storage, I saw things like regular backups of an error phrase "You have no access to this database". Guess what happened when the team accidentally dropped the database?

Unless you're doing regular restores, you don't have a backup. You have hope. So yes, doing backups in a way that gives you some form of guarantee isn't exactly cheap.

[hot_gril]

Or we do have everything in the backup, but the restore process isn't worked out. Someone is losing a weekend writing hacky scripts, and every SLO is being violated, if we ever have to use it for real.

[fishnchips]

Or you have everything in order but the encryption keys are gone.

[account42]

That case is still a lot better than no backups and having to tell your customers that the data is gone.

[wombatpm]

Make them test their disaster recovery plan. One organization I worked for took three tries before they could even simulate disaster their recovery process.

[Zababa]

I wish this kind of warning would come with a link to a website like dobackups.org that would document how to actually do them and test them. Something like a front page with different "profiles" like "Windows user", "Web application", "Database". Add to that some more precise docs, links to different solutions, maybe a bit of transparent sponsorship from backup companies, good practices like "how often to do backups", more specialized info for different use cases like family photos, legal documents (with user-contributed infos by countries).

I would like to do backups but it's already not really easy, even harder when you add the whole "figure out how to do backups" to it.

[Max-q]

After the Salesforce takeover? The one in 2010? So Heroku was good in like a year!?

[strix_varius]

Ex-Herokai here. In some sense, SFDC didn't start taking over H until several years after we were acquired. We had our own CEO, office, mission, org structure, annual meetup, etc.

Then, year by year, those things were taken away. Eventually, circa 2018, investment in the core H product started being reduced and individuals and teams started being reorged to focus on Salesforce priorities. H became a "keeping the lights on" job. Folks who wanted to keep pushing left to work on GitHub actions or on render.com, netlify, other places still striving for great devex.

[hot_gril]

Heh. I always loved Heroku and have complained that Salesforce is messing it up recently, but turns out the first time I used Heroku was like 5 years after the acquisition. I think Salesforce just started inserting their own branding more recently.

[mst]

At a guess, Salesforce bought it hoping it would take over the entire space rather than merely having the capacity to be one profitable provider among many, and lost interest once it became clear it was going to be the latter rather than the former.

(guess based only on having watched things unfold from the outside without ever being a user/customer so please do add salt to taste)

[hot_gril]

As a customer with no knowledge of the inside, this is all I've noticed, which probably misses some things:

1. GitHub integration breach, which wasn't handled well but luckily didn't affect me.

2. No more free-tier DBs :(

3. Salesforce logo, which ofc doesn't matter.

Hearing vague negative things here makes me nervous about the future because I really like Heroku and don't want to be stuck using AWS directly.

[mst]

A service can run in a state of if not benevolent, then at least ambivalent, neglect for quite some time without it being a disaster for the users.

I would suggest you start poking around at possible alternatives just in case, but not as yet with any great sense of urgency.

If you Ctrl-F for crunchydata in this comment section you'll find an employee of theirs talking about their postgres hosting and listing a bunch of services their customer use for the other parts of the puzzle, and have hired enough heavyweight core contributors and active and clueful community members that I think it's reasonable to say they're -good- at postgres.

I would say that what I expect to be most likely to happen is gradually increasing prices and gradually degrading service (the people still working on it do seem to care but I don't know if there are enough of them left to avoid bitrot setting in and even if there are today, in a year or two there may not be) so having a plan to move in an orderly fashion but not executing it yet seems like the wise approach.

Neither panicking and moving in a fast and risky way now, nor waiting until (if it happens) things go from aggravating to actively intolerable and moving in a fast and risky way then, are likely to be particularly good ideas.

OTOH, being prepared to migrate elsewhere in an olderly fashion if the cost/benefit calculations tell you it's time, re-running those calculations and double checking your plan every so often, and continuing to enjoy the service in the mean time, seems like a reasonable, responsible, and overall relatively pleasant way forwards.

(disclaimer: I am not a Heroku user myself, but I think the general principles almost certainly apply here just fine, and I'd certainly be comfortable giving the same advice to a consultancy client at work, so without being foolish enough to claim I'm definitely right I'd suggest the above analysis is at least worthy of giving serious consideration to)

[hot_gril]

Luckily I keep to basic ways of hosting things, so if the time comes to move, I should be able to. Replicated NodeJS web backend with a Postgres DB is gonna be supported in tons of places. No special logging, just stdout. Some cronjobs an outbound HTTPS requests, that's typical. The most unusual thing I do is open an SSH tunnel to some Linux server if there's a special long-running process I need to call RPCs on.

[kaushikc]

Don't mind my curiosity but I wonder if you do offsite backup of your emails or just take it for granted that Gmail is a reliable enough service ? It is very important to me but I never even considered backing up my Google takeout data. The point I want to make is that service reliability is a critical factor when considering what to backup. It was expected from Heroku to do the right thing and be conservative in their approach.

[rurp]

Anyone using Google products should absolutely backup their data regularly.

[TimWolla]

Personally I sync the emails in my Gmail account to my server at Hetzner using imapsync [1] every once in a while. The data on that server is backed up off-site at rsync.net using Borg. I'm intentionally not using Hetzner's storage boxes, because I want my backup to be entirely independent, e.g. if my Hetzner customer account is closed down for whatever reason.

[1] https://imapsync.lamiral.info/

[codegeek]

I hear you. I didn't do it for a while but I did 2 things more recently:

1. Setup IMAP backups locally on my computer and everything is backed up on computer to 3rd party backup tool

2. Use Google Takeout from time to time (it is a bit weird at times though).

[lamontcg]

Backups are apparently "too 20th century" for today's cloud-focused devops, you can just trust your entire business to someone else's procedures and if it all disappears in a puff of smoke, so do you.

(or just maybe we shouldn't have thrown out all the sysadmins with the bathwater)