[dzikimarian]

Apparently we're doing the impossible for over 12 years now. Who knew?

Some people act like it's some kind of black magic. It's not. We've some customers in our DC and some on AWS for various reasons. AWS isn't less problematic. AWS is about 10x more expensive. Both on prem and cloud require people familiar with them and cloud-engineers are in no way cheaper.

Only meaningful problem is that on-prem requires some up front cost&time. That can be mitigated by leasing and other means, but indeed can be an issue for small businesses.

[chongli]

Both on prem and cloud require people familiar with them and cloud-engineers are in no way cheaper.

I think the real story is a bit sordid: office politics. On-prem and cloud are different skillsets. Companies that have been around for a while can end up with both on-prem and cloud experts who end up competing with each other, often on separate teams. Throw in some slick consultants from Amazon who are able to bend the ear of the VP and you've got a real problem. From what I've seen it doesn't end well for the on-prem team!

[hirako2000]

I concur. Many VPs bend ears so easily you got to wonder. do they also get invited to some private dinner by those so called slick consultants, who pay the bill in a rush and leaves after forgetting some thick envelope on the table.

[chongli]

It’s a story as old as time itself. IBM has been doing it at least as far back as the 60’s. Fancy consultants who know the tech and also know how to sell and make themselves seem way smarter than the VP’s reports. Do one slick presentation and the VP is asking his team “why didn’t you guys come up with this stuff?”

Next thing you know these multi-million-dollar contracts are signed and the existing teams are just shaking their heads. The smart ones have already put out their resumes and started interviewing elsewhere.

[hirako2000]

I'm not one of those smart ones. So many things I was seeing for years make much more sense now.

It's an old story but I guarantee you many don't know about it.

[icedchai]

If you have a large enough on-prem infrastructure where you are using automation and not manually configuring everything, the skillsets will have a lot of overlap.

[overstay8930]

Cloud engineers can do the job of 4-5 on-prem people. Our AWS devs don't need to be BGP or ZFS experts, they just need to be AWS experts.

[rrrix1]

Hilariously ironic, with a sufficiently large cloud footprint, things like BGP (and more/other internetworking protocols) and OpenZFS become required skillsets. I have firsthand experience of this. :)

[FireBeyond]

Yeah, there was an amusement there. I've definitely had to understand BGP to configure cloud VPC setups.

"They just have to be an AWS expert".

Right. They just have to be experts in: EC2, S3, Aurora, DynamoDB, RDS, Lambda, VPC, LightSail, Athena, EMR, RedShift, MQ, SQS, SNS, ECR, ECS, EKS, ElastiCache, CloudWatch, CloudTrail, IAM, Cognito, and a few more. No big deal.

[dzikimarian]

Well our on-prem team doesn't need AWS pricing calculation and optymization expert, so there's that :-)

[owenmarshall]

AWS pricing and optimization is just capacity planning, which doesn’t go away if you run on prem - it just looks different, with longer time horizons & financial implications.

“Will my data center run out of floor space & I need to expand?” (years+)

“Will I have enough cooling & power to support the new racks we need?” (6 months+)

“When do I need to get the server order out to ensure we meet our capacity needs?” (6+ weeks)

Every one of those are capital expenditures, so line them up with the annual budget cycle - be sure to keep enough spare capacity to be responsive for last minute asks.

Don’t think my intent is to romanticize the cloud, either. It’s not better, nor worse, just a different way to manage things.

Of course if your company is sufficiently small, do whatever you know and can do quickly - customer acquisition will be more important than debating the cost of either infra in aws or a colo’d server or two in some racks somewhere. But the complexity doesn’t go away if you go to the cloud, OR if you are all on prem. TINSTAAFL.

[BackBlast]

2015 called and wants it's hype back.

[falserum]

Cloud clearly makes sense for:

- small business with at least some reliability expectation, and little to none IT expertise

- huge workload requirement volatility

- having someone else to blame

- solution is already working in cloud, with teams being very comfortable there, and perceiving on-prem as “enemy” (analogy: forcing devs to rewrite stuff from haskell to java)

- that extra cost is small budget line for you

On the other hand, it does not make sense to go cloud, if you are sufficiently big and already have on prem solution and expertise in house. (Extreme case: google, does not use aws for its main load; this upper threshold I wager is couple of orders of magnitude smaller)

[ipaddr]

Cloud doesn't make sense for small business. A vps would. If you are spending less than 100,000 you probably don't need it for your 10,000 million or less daily visitors

[Turing_Machine]

If you're spending less than 100,000, you almost certainly aren't spending enough to pay salary + benefits for a sysadmin.

[BackBlast]

Why does everyone always jump to this fiction?

You probably aren't paying a "cloud engineer" just to fiddle with cloud config full time with <$100k/yr spend. Why would you suddenly need a full time sysop to do the same capability level of on-prem? If you do 10 hours of "cloud engineering" a month to support an application, the same capability level of on-prem work is probably in the same ballpark. 5-30 hours. Yes, it can be lower than cloud. No, it's NOT suddenly 160 hours every month. Yes, it does mean you need someone who can wear the extra hat, cloud engineering skills are literally no different in this regard.

Internet sites ran on basement and closet servers for years, actual server rooms for larger stuff. The jokes were tripping over power cords and backhoes digging up internet lines were the causes of most outages. It's never been easier to run on-prem than today. It cost a fraction of even budget VPS providers like Hetzner.

For certain classes of applications, it's awesome. It's not everyone's cup of tea I'll grant. But if you are inclined to play with hardware or have someone in the org who does, and an extra 2-4 hours of downtime a year isn't that big of a deal (depending on general utility and network available at your chosen site). You can save tons of money.

[icedchai]

Exactly this. I've set up several VPSes for clients. They go for years without any maintenance because they don't want to pay for it. (They only bother when something is absolutely critical... like the OS is EOL and no more security upgrades.)

[ipaddr]

Why would you need additional people to manage a vps? The person managing amazon can very easily use cpanel because it is much simpler.

[MaxBarraclough]

> Extreme case: google, does not use aws for its main load; this upper threshold I wager is couple of orders of magnitude smaller

Not a good example in my opinion, as Google is also a major provider of cloud services. AWS isn't the only game in town.

I think Disney+ is a more interesting case. A quick google search turns up some articles saying their video streaming is powered by external CDNs. As I understand it, Netflix take the opposite approach and deliver all video data from their own Open Connect CDN, although they use AWS for other workloads (presumably including things like authentication, their recommendation engine, etc).

[dzikimarian]

Depending how small is small business, but I would probably go with VPS.

[coredog64]

Amazon retail runs on AWS, and I think we can agree that Amazon retail is reasonably described as “large scale”

[SOLAR_FIELDS]

Is that really a fair comparison though? AWS is a very weird argument to make because you could say that AWS is kind of “on premise” for Amazons purposes. Internally Amazon.com does not pay retail pricing or have the same level of support as third party end users. A better example would be looking at Jet.com/Walmart and asking if it runs on AWS.

[overstay8930]

Nobody who is big is paying retail prices, that's why saying "on premise is cheaper" is total copium.

As soon as you start factoring in discounts (i.e. bandwidth is nearly free at some point), the math of being on premise completely falls apart to the point you are paying more for licensing and support for the hardware than you are the entire lifecycle of your infrastructure in the cloud. It's just that bad to do it yourself.

[rstuart4133]

> licensing and support for the hardware

Sorry, but who pays for licensing and support of the hardware? I've never done that. You buy a Dell server or whatever, you pay for the 5 or 7 years warranty up front, you put in in a rack and you never literally touch it again until it's EOL'ed. If something breaks Dell touches it, not you. That typically costs around $500/year, albeit paid upfront.

But you usually don't do that. Instead you rent a dedicated metal from someone. The cost if of the order of $1000/year including some storage, no more to pay unless you exceed 100TB/year. A similarly configured AWS EC2 instance is $13,000/year, plus bandwidth, plus whatever other services you get sucked into. And you will get sucked into it because if you ask AWS about any problem (like say, monitoring why your bills are so high), the answer is invariably "use this paid service of ours".

You're kidding yourself if you think using AWS is cheaper than the alternatives. Those discounts you speak of are from an absurdly high starting point. I'm sure there are lots of reasons to use AWS or a similar cloud service, but unless you only need a lot of grunt for at most a few months price isn't one of them.

[BackBlast]

I ran the numbers the other day. For just compute with my particular load, my numbers say AWS costs 73928x more for lambda over my on-prem. Like for like is 1250x more. This is presuming the savings levels for being a big spender that I've heard about.

That's a lot of room to work with for some inconvenience.

[no_wizard]

This is a quirk of the business that is Amazon and AWS, as they started by selling excess compute and expertise, due to how Amazon was built as API first internally it was almost natural.

It’s in no way the norm for a smaller business.

[rented_mule]

This matches the public (i.e., non-Amazon) speculation I was hearing around the launch of S3 and later EC2. But not what I was hearing internally when I worked at Amazon. I was there when S3, the first AWS service, and EC2 were launched. I was working on what I believe to be the first Amazon (non-AWS) application that used S3 for storage. Getting that approved was not easy - all the same skepticism existed internally as externally (cost, availability, durability, security, etc.).

The story I was hearing internally was that it was too costly to scale infrastructure the way Amazon had been doing it, it was fragile, and the expertise wasn't keeping up with growth. So, set the bar a lot higher, and build infra that is big enough and flexible enough to be everybody's infra, and then Amazon's applications (e.g., retail) could run on AWS' excess capacity. Literally the opposite of what external folks were guessing. I believe they were completely physically separate data centers - even the physical location of AWS data centers were on a need-to-know basis internally (the internal lore was "under a mountain in Virginia" - this was years before Regions and Availability Zones). And any bugs in AWS could be worked out with outside usage before moving Amazon's applications onto it.

Also, Amazon needed the elasticity of AWS because of the nature of their retail business. At the time that the initial AWS services were being developed, a massive chunk of Amazon's traffic came during the holiday season. IIRC, something like half of the year's traffic and revenue, possibly more, came in November/December each year. That meant a lot of capacity was sitting idle most of the year. Selling that excess capacity would mean shutting AWS down every holiday season.

For a time, there was an internal mailing list that wasn't yet locked down that contained reports on S3 bandwidth usage. The growth rate was shockingly high. I would guess that within a year or two of release, S3 was using a few (at least) orders of magnitude more bandwidth than everything else at Amazon combined.

[no_wizard]

In broad strokes, the main point I was makings till stands though: AWS was deliberately made to back the demanding scale of Amazon, it was a bet on the future and the Amazon model as much as it was product service, and that did mean they built expertise and hardware up and sold that as a product none the less.

This still isn't the norm for most businesses, even big ones.

[no_wizard]

In many organizations the biggest appeal of services like AWS or GCP isn’t simply cost, it’s that the service is approved and therefore all the services of that service are approved and I no longer have to justify spinning up more compute or leveraging one of their more bespoke services like SQS (or wherever equivalent). It’s all just there, ready to be used.

It may not be true for you and where you work but this is a very real thing in a lot of organizations, where development teams want a quicker turn around on booting up services they need as the product evolves and having some control over how they act with each other. It (sometimes to negative results) opens up more architectural possibilities to solve problems

[dzikimarian]

I'm not saying using AWS is universality bad. Depends on your needs.

What I'm saying is some people are trying to portrait rolling your own k8s or on-prem as equivalent to rolling your own crypto - better left to the chosen ones with years of training in secret monastery. This is BS :-)

[tqi]

These types of debates always seem to go this way - one person saying one option works way better for them so the other option must be crazy, followed by another person saying the opposite. I'm not an infra guy, but my guess is the reality is both are choosing the right option for themselves, because is no one objectively "best" option. Just tradeoffs.

I also think if a company is unhappy with their current set up and thinks that switching from cloud to on-prem (or vis versa) is the answer, they're probably delusional because "the fault, dear Brutus, is not in our stars, but in ourselves."

[dzikimarian]

Completely not my point. I don't say that using cloud is crazy(we're using it sometimes). I'm saying that opinion that on-prem is much harder than AWS is IMO wrong.

Requires some planning ahead of time - yes. Harder - not really.