[pdimitar]

Currently I work in a company that's very cost-conscious about hosting. They are generous in salaries but are mercilessly practical about IT costs and the abilities of computers -- especially the latter part is something that most of the programmers today have almost completely forgot.

I have 3 spare laptops lying around; weakest of them is with a Celeron J4155 and I have put a web app with no caching on it (it does have a persistent DB) and hammered it with my workstation until it finally started giving up at ~2500 req/s (Elixir/Phoenix stack). Again, that's a Celeron J4155 with a SATA III SSD in an M.2 factor (so disk speed caps at 550MB/s at best; usually 400-460) and 12GB RAM. Most programmers wouldn't touch such a machine.

I imagine I can buy 2 more of these laptops and make a completely replicated 3-cluster of the entire stack of our company and the slowest requests (on admin UI where we have a lot of SQL JOINs) would likely never go above 200ms. That totals at about 600 EUR (yep, I bought the laptop second-hand for 200 EUR). Then 500 more EUR for a good UPS to plug the laptops and my routers to. Boom 1100 EUR and several weekends later I can likely charge my own employer for hosting at 100 EUR a month for their entire infrastructure and I would likely still be ripping them off even with that.

The only real cost is human time and energy invested in making it work. But for most companies that's not a 24/7 fight so that cost is fairly low. You can do it twice a year and you're likely never going to have problems.

So yep, I am completely with you here (if my rant didn't make it obvious). Infrastructure costs are already being heavily optimized by companies out there.

[lotsofpulp]

> The only real cost is human time and energy invested in making it work. But for most companies that's not a 24/7 fight so that cost is fairly low. You can do it twice a year and you're likely never going to have problems.

If this were true, what made the cloud providers popular in the first place?

[pdimitar]

Outsourcing culture. Nobody wants to nurture talent -- that would also mean to invest in relations with your employees and not alienating them. Shareholders prefer to sacrifice a little more of their profit so they deal with less potential problems. And the pull of the idea that every human in the org must be an inter-changeable cog is too strong (even if that idea continues to be absurd, and always was).

Hosting apps in the cloud was a fair exchange 10 years ago because operational tooling in general was more immature. Nowadays it's much easier to self-host many pieces of software though.

[colechristensen]

This isn’t it, you might not know or if you haven’t been in an environment like it.

Without a cloud you’re always running up against limits, out of power, out out cooling, out of rack space, out of hardware. You get new resources by adding to wish lists and seeing if the end of quarter budget will agree with your request which might be filled in a few months, maybe next year, often never.

You hoard hardware that ends up doing nothing most of the time so you have it when you do need it. Management spends a lot of time and energy managing the datacenter budget.

With cloud you get what you want without asking too much and management periodically spearheads savings efforts to show off, but ultimately usually spends a lot more than they would have otherwise with less friction.

A big part of cloud adoption, according to my theory, is getting executives out of the way of computing resource needs and freeing up their time to fill with something else like bothering employees for more status updates (which are easier and require less skill).

[oceanplexian]

> Without a cloud you’re always running up against limits, out of power, out out cooling, out of rack space, out of hardware.

I grew up in this era and keep hearing this repeated but it simply wasn’t true. Enterprises would plan ahead and buy enough hardware for years and it would work fine until you bought more. The myth that you need to scale your infrastructure 10x in a day doesn’t apply to 99% of enterprises, and even if it did it’s probably a result of bad planning on the part of leadership. As a result of the current paradigm businesses end up renting servers at a substantial markup for fairly obsolete hardware.

[zrm]

In general it's really the opposite. In 2004 you needed a rack full of $3000 servers to run your medium business. Now it's two physical machines using 5% of the power to virtualize everything that used to run on two dozen.

Over a given period of time, computers get faster/cheaper by more than most businesses expand. When you need to expand, buying a newer, faster machine may cause you to save money because the faster machine uses less power than the existing one.

[pdimitar]

You are talking big businesses here, I've seen people just ask the CEO if they can buy 3 brand new servers and him agreeing, verbally, and the servers arrived next week, and two weeks later were completely setup and were useful.

This lasted for 11 years and only stopped because two of the 5 senior engineers retired and because the company was bought a few weeks earlier.

So again, don't look at this through Silicon Valley lens. Most of the companies in the world have a very different mold compared to SV.

[heisenbit]

Then there are service contracts to make surr the systems keeps running with little downtime. Which may be impossible to get for 11 year old equipment. Then there are pesky details like needing a disaster recovery site.

Professional hardware is expensive and server h/w is a small part of it.

[sangnoir]

> Without a cloud you’re always running up against limits, out of power, out out cooling, out of rack space, out of hardware.

I bet most customers of cloud services are not in a high-growth phase, so this is scenario most organizations aspire to ("What if we suddenly got popular?" is a fantasy that's hard to disabuse someone of internally, if you want to be known as a team player

> Management spends a lot of time and energy managing the datacenter budget. With cloud you get what you want without asking too much...

I fully agree, this is the core reason why most companies gravitate towards cloud: management abdicates control of costs to engineers, resulting in less friction - but its OpEx, not CapEx, so the bean counters are chilled about it. If the same low-friction approach were applied to DC equipment, you'd get similar results, but cheaper.

[hallway_monitor]

I used to be a big proponent of self-hosting. The raw hardware cost makes it look like a great deal. However in my experience hardware is less than 5% of salary cost and having enough admins on hand to make your infra reliable is usually going to end up costing you more in the end.

[ipaddr]

Don't some of those salary costs transfer over to the cloud. Someone still has to manage it

[Jensson]

You need a lot less people per computer when you have millions of computers like big tech has than when you have 10.

The cost of these services is not because big tech has to use that much to run them, but because big tech would make less money if they lowered prices. AWS generates tons of profits, why lower that for no reason?

[kazen44]

> Outsourcing culture. Nobody wants to nurture talent.

Also, outsourcing moves the blame to someone else if things go wrong. (and things in infra go wrong nearly constantly).

The problem with this kind of thinking is ofcourse, that there is no risk taking and innovation in suchs an organisation..

[nthj]

AWS lets me provision servers all over the world at any point, which is required by customers both for compliance and for latency SLAs. Even billion dollar companies are going to find that a huge lift to build out themselves, and they have opportunity cost while they try to do it.

[AzzieElbab]

What about scaling down? Once you bought hardware for self hosting you are stuck with it.

[pdimitar]

What about it? We're talking small businesses here, this will not be a warehouse full of racks worth of machines to try and sell. We're likely talking 3 to 10 PC-sized machines in a closet or storage room. You can sell those on 2nd hand market pretty quickly and even if you sell them at 30-40% loss you are extremely likely to have already paid off the other 60-70% during the machines' service time.

[AzzieElbab]

old laptop is fine then :)

[naasking]

The costs the OP described are pretty minimal.

[JBlue42]

You never know. While working in an ITAM role, I've had a manager point out a line item asking why we need 30 DisplayPort cables. This was in an org with multiple sites, 2000+ employees, using dual monitor setups both in house and remote. Whether physically, or in the cloud, companies whose primary business isn't tech-related see the IT side as an annoying cost center rather than a cost of doing business in the 21st century.

[Lamad123]

This selfish outsourcing culture is killing all the Gilfoyles out there!

[BbzzbB]

I'm curious as a hosting and cloud noob, do you have an estimate how much that would cost monthly or yearly if hosted on a cloud?

[pdimitar]

Not really, because AWS costs aren't transparent. You can't drill down properly and even where you can, the prices are disappointingly high.

It gets really tempting to setup a backup/failover node on one of my spare laptops lately...

[lotsofpulp]

Interesting. I am not intimately familiar with the subject matter, so I will be curious then to see how the next 10 years play out.

[rossmohax]

Beancounters love OPEX and avoid CAPEX

[kazen44]

beancounters are not in charge of the direction of the company. In a lot of cases, it makes a ton of sense to go for capex vs opex.

My old employer had a strategy which basically boiled down to owning everything inside this company expect for the coffee machines and cleaning crew. His reason being? This made it possible to run on very thight margins when the economy takes a downturn without having to scramble for money because of leased/loaned equipment etc.

In the 2008 crisis, this is how he stayed afloat with his company, and even made a pretty profit during a time of crisis too.

[lotsofpulp]

Surely the beancounters can also see the effects of capex on the valuations of the cloud providers.

[ethbr0]

If they're publicly traded, no one's going to give Bob's Consolidated Widgets a 65x PE or 2x+ PEG ratio, even if they have the best IT strategy in their market.

[abecode]

Curious, can you explain why? In my experience (just one company where I had some interaction with accounting), there was a preference toward CAPEX. The costs of building a product, ie software development, was prefered over OPEX, in this case, analytics. Since I was doing both development and analytics, they preferred me to account part of my time as CAPEX. Not sure if this is normal but for context it was a non/pre-public company.

[xunn0026]

OPEX is just an expense in the current fiscal year. CAPEX depreciates over N years. This alone means that a one-time payment is much simpler on the books.

I don't think your salary counts as CAPEX, it's a normal monthly expense for the company.

[Nextgrid]

Keep in mind that most big clients of cloud providers are startups running on VC money. It's not their money and they don't care about wasting it, and VCs don't particularly care either.

This has now created an entire ecosystem of developers and "devops engineers" (what we used to call sysadmins) who know little beyond the cloud and have to keep using it for career-related reasons. This in turn pushes companies to use the cloud as finding talent for old-school on-premises infrastructure is difficult.

[mindslight]

Infinite scale up (direct fan-out to mass market), high margins (able to overpay for commodities), and outsourcing production work to stable businesses are all facets of the Hollywood business model that Surveillance Valley adopted. Once the concept was socially proven, then nobody gets fired for buying IBM.

[nicoburns]

That human time happens to be quite expensive.

[pdimitar]

Not in small companies.

[ghaff]

It's most of the cost at small companies. A lot of people underestimate the total cost of having employees perform tasks at unpredictable scale with unpredictable problems versus just writing a check each month.

It's true that cloud providers aren't reliably cheaper as some assumed at one point. But there's something to be said for cutting out the fully burdened costs of some number of employees (especially given everyone is saying tech talent is expensive and scarce at the moment) and just letting AWS deal with it even if you could theoretically do it cheaper yourself.

[13of40]

Not only that, but if you have a business that actually makes, sells, and ships something, your AWS bill is probably a tiny fraction of your expenses. Switching to physical servers would be a little like getting solar installed on your house - great in theory, but if the most expense it will displace is $150/month, is it the right problem to focus on?

[nicoburns]

True. Although I’d argue most companies are small enough that AWS itself represents more overhead than is optimal. They’d be better of on Heroku, or even just a single VM.

[rcpt]

What if the cheap guy quits?

[pdimitar]

I'm aware of the risks. That's why I said a culture of long-term relationships must be fostered if you want to pull such operations off.

Where I work right now people don't just quit two days later. They like the company and if they feel they want to go someplace else they're not being difficult about it. They cooperate in passing down their knowledge to colleagues before they go.

[GabeIsko]

You are vastly overestimating the IT capabilities of most companies. The human time and energy cost that you handwaved away is a real killer because that is expert knowledge capability that is highly valued by the market. Maintaining an IT staff and infrastructure to support a large organization that has to perform any kind of reasonable SLA for services eventually puts you in direct competition with Major Cloud Providers at scale. A scenario where you can outcompete them with their large customer bases becomes very difficult to imagine under current technological and market conditions.

It's not a matter of outsourcing culture. Cloud providers really do provide computing in a cost efficient matter since they practically provide IT infrastructure at wholesale, and it produces an anti-trust risk because, as you pointed out, the actual hardware that can make it go is not nearly as expensive.

[pdimitar]

I am not handwaving anything away. We're a company that has less but very strong programmers (we don't even have DevOps / sysadmins, our CTO is doing it and I'll likely be taking part of his responsibilities at one point).

And we don't have the requirements of most big Silicon Valley companies that you reference. It boggles my mind why so many commenters IMMEDIATELY assumed we're a hyper-growth startup or whatever. We're an expanding data provider and our growth is super predictable, and our set of requirements changes like once a year.

So really, not sure why you and others keep repeating things I already agree with. The cloud wins from one point / scale and on, absolutely; otherwise it wouldn't ever take off. I am saying that people give up way too quickly and run to the cloud long before they need it -- that's all.

[GabeIsko]

If running your tech stack on local hardware is right for you guys, than more power to you. That is also what works for me for my personal and hobby computing requirements, where I maintain a home machine. But there is kind of an implication that most people who start developing on cloud services are just clueless programmers who never learned the basics of system provisioning.

Even with the specs you mentioned, you can rent a lightsail server that is comparable on AWS for like 80 usd a month, so I would push back against the idea that using cloud services in pretty much any scenario is a purely technical decision.

Part of the context of discussing things this way is the original article: do the prevalence of service companies signal the end of big tech? I would say no, because the only way to not be reliant cloud service providers is to do what you describe and compete directly with their infrastructure services by forming your own technical infrastructure. But the logical conclusion to doing things this way is that eventually whoever can control more physical infrastructure has an advantage. What if Amazon is able to replicate your service offerings? They would be able to immediately able to offer their version using their cloud services at a fraction of the price because they can amoratize the cost of computing over their cloud business. Then, when they have eliminated you as competition, they can recoup everything by increasing the price. This is a HUGE anti-trust concern, and one that all tech companies should be at least a little bit concerned about in the long term. It doesn't go away because you can have an (in your case more expensive than lightsail) up front investment to own your companies computing resources.

[pdimitar]

If they can do it at the fraction of the cost then I'd use it. Trouble is, AWS pricing is hard to parse, especially post factum. In my company's case, the leaders are not happy with the lack of billing transparency.

[GabeIsko]

I think that's not only a good reason not to use AWS, but also the fundamental problem with the position that cloud service providers pose. It would be much healthier for companies and organizations to directly own their own computing resources, rather than effectively leasing them from one of three cloud service providers, but it is absolutely more of an investment for companies to make.

[pdimitar]

Sure, agreed. I am on the opinion that Amazon does not have to be that monopoly-oriented but alas, humans being humans right?

Give me an AWS service with predictable billing and ability to put hard limits so you don't have your credit card charged for surprising amounts if you get bursty unexpected loads, and I'll be all over them without thinking of on-prem hosting.

[dash2]

But you yourself admit that you have an unusual setup with a few very strong programmers. Few are like this even in Silicon Valley. As for other industries, they're not even close. So your company's setup won't transfer well to the rest of the world.

[pdimitar]

I would not be so sure of your last. The SV mold is the outlier but most USA programmers can't see it because they don't work for non-US companies. Everywhere I was all around the EU in my 20 years of career so far, the SV way of doing things has not been the norm.

So it pays off to be wary of filter bubbles. The SV way of doing things assumes a ton of VC money to burn, something that, to this day, is still not a wide trend in the EU (even if it happens here and there).

Plus that company's setup is not unusual in any way per se; the only "unusual" thing is "we don't run to the cloud at the first IT problem". A ton of other companies are like this, I'd bet even in the US.

[capitalsigma]

What happens when your local internet connection goes out and you can no longer serve traffic for your business? What if there is a power outage? Who will spend the time replacing disks as they fail? Where is the data backed up to, so that a failed disk doesn't lose business data?

[xanaxagoras]

I like where you're going with this but you could take it a lot further with little expense. Amazon and MicroCenter have a huge stock of refurbished HP mini PCs with i5s, 16gb of ram and an M2 slot for about $350. Throw in a small m2 ssd in each (~$50) and keep your UPS, that's about $1700 you have a decent cluster for an efficient stack like phoenix.

[pawelmurias]

How much money would your company save by hosting on some old laptops instead of in the cloud?

[pdimitar]

5 digits of USD per month. Whereas normal on-prem hosting can be started with almost no dollar cost, tomorrow.

Right now they/we can't do it though, the talent is there but that talent is super busy reworking software that's no longer well-adapted to their new requirements. We'll see if the topic becomes more relevant in the future.

[thow-58d4e8b]

Not to forget - there are more choices than big-cloud and host-everything-yourself.

One can rent VMs from Hetzner at 1/3 the price of AWS (and 1/10 the data transfer fees). No managed services, like Dynamo or S3, but will full convenience of on-demand scale-up

Or, for mid-sized corporations - buying a rack of computers, wiring them together, and installing OpenStack to make an internal cloud - that isn't some black magic wizardry either

[pdimitar]

Oh yeah, completely agreed. As mentioned in another comment we might just go Hetzner and buy a singular server in another provider for backup and failover. We'd likely be just fine with 6-7 of the smaller Hetzner servers, too.

[pawelmurias]

It seems crazy that you are paying that much for hosting something that can be self-hosted on an old laptop.

[_alex_]

that works as long as there are enough of you for every team/app/service/whatever to build and operate all that around the clock and without bus risk.

A lot of these threads end up talking about "scaling" from a pure hardware perspective. Once you start talking about medium-to-large companies, the real scaling headache is people, process, and organizations.

[pdimitar]

Not sure if I made myself clear but I definitely meant my above as aimed at small businesses (which my current employer is).

Obviously from one scale and on the cloud is very much worth it. IMO the discussion has to be shifted to "before which point you can easily get away without the cloud?".

[makapuf]

At some point, an internal cloud starts being interesting and gives a clear org boundary, using cloud-based software and interfaces.

[pdimitar]

That's true as well. It all depends on how much is the employer willing to invest in their employees in the end; if they foster longer-lasting relationships then many interesting internal projects become feasible or even desirable.

[ghaff]

Those invested-in employees will probably still leave, especially if they're very good and you can't match big tech salaries. And, in any case, you need redundancies for when they're on vacation, out sick, etc. Not to say you shouldn't develop in house competencies but you should do so deliberately because there are probably a lot more costs and risks than are obvious.

[pdimitar]

We're not blind to the tradeoffs. It's a long process to both change culture and nurture the necessary skills in your staff, AND take care of having human redundancy.

My point never was to idolize on-prem; I have agreed in other comments that from one point and on the cloud absolutely wins.

I'm mostly pointing out that there is a lot you can do before bowing your head to Amazon and accept $30_000+ monthly bill for infrastructure that I can fit in my 1 square meter food closet and which would likely cost me (or a few other experienced backenders) 2-3 weekends to setup.