[bird_monster]

> That looks way cheaper, but then you have to do the engineering and the operational support yourself.

In my experience, this is the piece that engineers rarely realize and that is actually one of the biggest factors in evaluating cloud providers vs. home-rolled. Especially if you're a small company, engineering time (really any employee time) is _insanely valuable_. Valuable such that even if Airflow is cash-expensive, if using it allows your engineers to focus on building whatever makes _your business successful_, it is usually a much better idea to just use Airflow and keep moving. Clients usually will not care about whether you implemented your own version of an AWS product (unless that's your company's specific business). Clients will care about the features you ship. If you spent a ton of time re-inventing Airflow to save some cost, but then go bankrupt before you ever ship, rolling your own Airflow implementation clearly didn't save you anything.

[tpxl]

We used to have on-prem redis and a devops engineer to manage it, then we moved to redis in the cloud and had a devops engineer to manage it.

Saying that in the cloud you don't need engineers to manage "operational support" is the biggest lie the cloud managed to sell.

[glogla]

If you just run redis on virtual server in cloud, you're not replacing the devops engineer who manages redis. You're replacing:

- the network engineers who manage switches, firewalls, routers, nats and connectivity

- the people who manage on-prem hardware, install new servers, replace failing servers, and switch broken disks for working ones and install base os using some ilom

- the whole process for ordering that hardware and getting it delivered on time - including knowing when it's going to be necessary

- if your on-prem had virtual servers, the people who manage and operate vmware

- if your on-prem had SAN, the people who manage and operate that SAN (and buy new disks and take care of capacity planning)

Some of those things you still have to do - for example configure firewall, or say "I want to take a snapshot of this virtual disk drive" but instead of doing a difficult technical thing, you can do it in web ui - but you still need to do what to do.

And of course, if you never had SAN and virtual servers and two data center with your team managing the interconnect between private networks, there's a lot of stuff that the Cloud could give you that you probably don't need.

Now if you move to managed redis, you're also replacing the person who installs and patches the linux the redis runs on, and the one who installs, backups and configures the redis. And you get the redis on button click, so if you suddenly need three more, you're also replacing the person who automates building redises.

You are right, that it is not that the operational support is just gone. Some of it is gone. Some of it is replace by doing cloud stuff (like thinking of buying reserved instances instead of actual physical servers). Some of it is just more efficient.

Now if any of this doesn't fit you're use case because you have too small or too large scale, then the Cloud is of course a bad idea.

[bird_monster]

You've just wrapped a very specific definition of "Operational support" into a very vague term.

From another post in this thread: If you want to deploy distributed stream processing like Apache Kafka, but do not want to roll it yourself, you can use a tool like AWS Kinesis or Azure Event Hubs.

You still need "Operational Support" to manage EH/Kinesis, but generally it's closer to the type of "Operational Support" that can be provided by a general backend software engineer, as opposed to a DevOps/Infrastructure-specific dev. By using a Cloud (Managed) service, you're removing the need to manage:

* Uptime management * Scaling * Custom-multi-AZ

And probably a lot more. Sure, you've still have to actually handle events appropriately, but you have to do that either way.

[glogla]

I agree.

The only caveat is that this goes for founders or engineers who are financially tied with the company success. If the engineer just collects paycheck, they might prioritize fun - and I feel that might be behind a lot of the "reinventing the wheel" efforts you see in the industry.

Or maybe I'm just cynical.

[Silhouette]

Especially if you're a small company, engineering time (really any employee time) is _insanely valuable_.

This is true, but it is balanced by the fact that uncertainty can be insanely expensive. And diving into complicated cloud infrastructure with a small business, if you're not already an expert on it, is a very uncertain endeavour in terms of whether you'll get everything set up right (and not find out otherwise at 3am when it turns out your redundancy wasn't, for example) and what everything will cost. By the time you have either become an expert yourself or hired someone who already is, your costs have already increased significantly too, for exactly the reason you've just stated yourself.

[bird_monster]

> And diving into complicated cloud infrastructure with a small business, if you're not already an expert on it, is a very uncertain endeavour in terms of whether you'll get everything set up right

I do not agree. The entire point of using cloud offerings as opposed to rolling your own, is cloud offerings are usually several orders of magnitude easier to configure. Using Event Hub, as an example, means that you're getting a similar experience to Apache Kafka, but without having to scale/configure everything yourself.

Sure, you have to become proficient with Event Hub, but becoming proficient with EH is probably 1/100th of the difficulty as becoming proficient enough in Kafka to support the same scalability/workload.

[Silhouette]

All I can say is that this hasn't been my experience. Setting up a single server is much the same whether it's on-prem or at a colo facility or some VM in the cloud, but the amount of added complexity to set up non-trivial networking and security and redundancy/failover and backups and all that stuff in the cloud is far more complicated -- if you don't already know how to do it -- than just setting up a few servers directly. The only exceptions IME tend to be the most basic and essential services, like effectively infinitely scalable storage and managed databases. Not coincidentally, I suspect, these are the kinds of cloud services that almost everyone uses, and often (as this very HN discussion demonstrates) the only ones.

There is still considerable value in the on-demand nature of cloud hardware, instead of waiting for stuff like physically assembling new servers and then shipping them and then installing them in the rack all before you can even start setting up the software, but IME the simplicity emperor has no clothes. Just look at the number of anecdotes about even quite large and well-established businesses that have had systems go down because they didn't fully understand AZs and failover arrangements, or have been landed with some crazy high bill because they didn't fully understand all the different things they were going to be charged for, or have been breached because they left some S3 bucket open.

[glogla]

Watching your discussion I think the truth might be somewhere in the middle - the Cloud can do things for you but you still need to learn how to use it.

So once you know your way around, it can be a force multiplier but learning cloud can be as much work as learning how to do it on your own.

Or said from a different angle, it help you outsource (part os) operations but it does not actually help you to engineer and architect things correctly, and you can shoot yourself in the foot pretty easily.

Personally, I think the cloud is most transformative for large companies with dynosaur internal IT, where it brings a lot of engineer self-service into the picture - where I work I can have RDS in minutes, or internally provisioned on-prem Oracle in a week, and the Oracle ends up being more expensive because 24/7 support has to be ordered from a specific list of vendors ... but that's not going to be the case in agile company with strong engineering culture.

[Silhouette]

I suspect much or all of this is true, including the observation about large companies with dinosaur IT. The comment I originally replied to was specifically talking about the environment in small companies, which is the environment I'm most familiar with, and my comments should be read in that context.

In my own businesses or the clients we work with, we too could add RDS quickly for something we were hosting on AWS. On the other hand, we could also spin up a pair of Postgres instances and configure streaming replication quickly if we were running on-prem or colo. Each approach has its pros and cons, and we'd look at each situation on its merits and choose accordingly. But IME, it's more like choosing from a restaurant menu depending on what looks most appealing at the time. The way some people talk about cloud, it's like they see it as choosing between a gourmet restaurant with a Michelin-starred team and the local burger van.

[bird_monster]

> but the amount of added complexity to set up non-trivial networking and security and redundancy/failover and backups and all that stuff in the cloud is far more complicated

This complexity exists either way, is my point. Whether you're managing your own servers, or using barebones cloud VMs, or using a bunch of cloud fanciness, the complexity you just defined still exists. And if that complexity is a constant, why is it only being used as a negative against cloud services?

> Just look at the number of anecdotes about even quite large and well-established businesses that have had systems go down because they didn't fully understand AZs and failover arrangements, or have been landed with some crazy high bill because they didn't fully understand all the different things they were going to be charged for, or have been breached because they left some S3 bucket open.

If your argument is "It's not better when done badly", definitely, I agree, because what is?

I guess, my overall point is that cloud-based infrastructure shifts your focus. Yes, you have to know how to configure cloud resources, but in 2021, do you think it's easier to find people with AWS experience, or people with custom in-house or colo server management experience?

[Silhouette]

The thing is, I don't think the complexity is even close to the same in the two cases.

AWS and similar services are an abstraction over hardware, software and networking all at once. There are well over 100 different services available on AWS alone. Just to get a basic configuration up and running, someone new to the system has to figure out which of those services they actually need, which is a barrier in itself given the obfuscated names they have.

Then you have much the same network and security details to set up as you would have for on-prem or colo infrastructure, but now with lots of non-standard terminology and proprietary UIs, which are so cumbersome at times that an entire generation of overcomplicated "orchestration" tools has been developed, each of which typically adds yet another layer of leaky abstraction.

Hopefully some time before this all happened you tried to work out what it was going to cost, and maybe you were close or maybe you are in for a nasty surprise because those handy managed services cost several times what the equivalent server + software would have cost either running on real servers or just on cloud VMs.

And if you fall back to that latter case as your safe default, you still get all the same issues to deal with as you would have had on your own servers and network, except that now you need to figure out what is really going on behind all those virtualised systems and quasi-geographical organisation layers before you can tell whether one unfortunate event could take down all the instances of any vital services you need.

In comparison, literally every small business I have ever worked in as a tech worker has had several people at the office who were perfectly capable of buying a switch or firewall or router and spending the few minutes required to configure it or buying a server and installing Linux/Windows and then whatever server software it needed again very quickly. Cloud systems can make it faster to deploy new hardware and connectivity, because you save the time required for all the physical steps, but after that the time and knowledge required to get a small network's worth of equipment up and running really isn't that great. After all, we used to do that all the time until the cloud hype to hold, and it's not as if that has suddenly stopped working or all the people with that knowledge suddenly left the industry in the past 5 years.

[bird_monster]

> The thing is, I don't think the complexity is even close to the same in the two cases.

Agreed (but probably on the opposite end as you)

It seems a lot like you've been scorned in the past and that's driving a lot of your statements now (which is totally fine and fair). I'm trying to bring up that, for every problem you've just defined, the literal exact same problem exists for colo/managed servers, except it is now also your problem to keep the lights on and the machine running.

> literally every small business I have ever worked in as a tech worker has had several people at the office who were perfectly capable of buying a switch or firewall or router and spending the few minutes required to configure it or buying a server and installing Linux/Windows and then whatever server software it needed again very quickly.

I'm sorry, if you believe that building and deploying production-ready server infrastructure is as easy as "Just going out and buying a switch and spending a few MINUTES installing linux" (emphasis mine) - I feel like we aren't talking about the same thing at all. Not even close.