[kkielhofner]

AWS/clouds aren't always the best solution for a problem. Often they're the worst (just like any other tool).

You don't provide a lot of detail but I imagine at this point you need to get "creative" and move at least some aspect of your operation out of AWS. Some variation of:

- Buy some hardware and host it at home/office/etc.

- Buy some hardware and put it in a colocation facility.

- Buy a lot of hardware and put it in a few places.

Etc.

Cash and accounting is another problem. Hardware manufacturers offer financing (leasing). Third party finance companies offer lines of credit, special leasing, etc. Even paying cash outright can (in certain cases) be beneficial from a tax standpoint. If you're in the US there's even the best of both worlds: a Section 179 deduction on a lease!

https://www.section179.org/section_179_leases/

You don't even need to get dirty. Last I checked it was pretty easy to get financing from Dell, pay next to nothing to get started, and have hardware shipped directly to a co-location facility. Remote hands rack and configure it for you. You get a notification with a system to log into just like an AWS instance. All in at a fraction of the cost. The dreaded (actually very rare) hardware failure? That's what the warranty is for. Dell will dispatch people to the facility and replace XYZ as needed. You never need to physically touch anything.

A little more complicated than creating an AWS account with a credit card number? Of course. More management? Slightly. But at the end of the day it's a fraction of the total cost and probably even advantageous from a taxation standpoint.

AWS and public clouds really shine in some use cases and absolutely suck at others (as in suck the cash right out of your pockets).

[tarun_anand]

100% agree. Most public clouds are ripoffs. We have spent 11 years on it and now thrown in the towel.

Go for some colocation facility where costs are predictable.

[dspillett]

It depends on your use case and internal infrastructure support. A lot of start-ups start on "cloud" when they have unpredictable needs and little immediate cash for kit & sys-admins (to manage more than the bare servers: backups and monitoring and other tasks that a cloud arrangement will offer the basics of at least, will need to be managed by you or a paid 3rd party on your kit). Later when things have settled they can move to more static kit and make a saving in cost at the expense of the flexibility (that they no longer need). Or they go hybrid if their product & architecture allows it: own kit for the static work, spreading load out to the cloud if a temporary boost of CPU/GPU/similar power is needed (this works best for loosely-coupled compute-intensive workloads, which may be the case here depending on exactly what they are trying to get out of ML and what methods & datasets are involved).

[zupa-hu]

This should be top voted. Buy the hardware and expect your costs to fall 10x.

[peterwwillis]

There are also more upfront costs (not just monetary), you can't scale quicky, and you lose all the managed solutions that make building things super fast and effective. Your hardware cost may be lower 10x but the operational and developmental cost will be higher as well as a limit on your business to grow.

A balanced approach is to only put the most expensive hardware portion of the business with the smallest availability requirement in colo, and horizontally scale it over time. Simultaneously use a cloud provider to execute on the cheap stuff fast and reliably.

[philliphaydon]

> AWS/clouds aren't always the best solution for a problem.

And when they aren’t always the best. It’s often because you don’t know what you’re doing.

It’s too uncommon for people to over provision. Or go with too many services when they don’t need to.

Like let’s have a database and cache service and search search. When 95% of the time they only need the database because it can do full text searching adequate enough and they don’t have the traffic to warrant caching in redis, and can do basic caching.

They don’t take advantage of auto scale groups, or run instances that are over provisioned 24/7.

I’ve seen database instances where when it’s slow they throw more hardware at it instead of optimising the queries and analysing / adding indexes.

The biggest cost of cloud providers is outbound data. The rest is almost always the problem of the Developers.

[rumanator]

None of your comments are relevant to machine learning applications, and all you do is throw blanket statements about ignorance. Your comments are very far from the problem and from being helpful.

[philliphaydon]

Nope. We have no information of the OPs setup, bill, or anything. This entire thread is based on assumptions. I common examples of developers screwing up and generating large bills. Explain to me how machine learning is any different.

Do we know if the instances used for MLing are running 24/7 idle until customers use them? Do we know if the utilisation is optimal for the workloads?

We know nothing. So claiming that cloud providers are not good is very far from the problem and not helpful.

[rumanator]

> So claiming that cloud providers are not good

The statement is not that AWS is "not good". The statement is that AWS is very expensive, specially for computational tasks, and there are cheaper alternatives around.

AWS is notorious for positioning their services as a way to convert capex into opex, specially if your scenario involves a SaaS that might experience unexpected growth and must be globally available. Training ML models has zero to do with those usecases. It makes no sense to mindlessly defend AWS as being the absolute best service around for a job it was not designed for and with a pricing model that capitalizes on added value on things that are not applicable.

[philliphaydon]

I never defended AWS as being the absolute best. I said high bills are almost always due to developers and not the cloud provider. Which you haven’t argued against.

As I said I have examples of how Developers often cause large bills.

And I explained why we can’t help with the OPs large bill.

You’re saying that with ML there is absolutely 0 way to reduce costs on AWS which is absolute rubbish.

[rumanator]

> I said high bills are almost always due to developers and not the cloud provider.

I feel that's where you keep missing the whole point. Somehow you're stuck on thinking that an expensive service is not a problem if you can waste time micromanaging and constantly monitoring expenditures to shave off a bit of cost from the invoice. Yet, somehow you don't register in your universe the fact that there are services out there that are both far cheaper and arguably better for this use case.

Therefore, why do you keep insisting on the idea of wasting time and effort micromanaging a deployment like pets to shave off some trimmings off a huge invoice if all you need to do to cut cost to a fraction of AWS's price tag is to.... switch vendor?

[geofft]

Nothing is stopping you from applying all those optimizations to on-premise hardware, right?

That is, I am not sure "public cloud, if you spend lots of effort to optimize it and ask devs to be careful, can be as cheap as a naive on-prem implementation where devs don't need to be careful" is an argument for public cloud.

[philliphaydon]

Well if your on prem then you’re probably bit more limited in what you can do. You can’t just go “let’s solve this with x” cos x doesn’t exist so you need to prevision it yourself and maintain it yourself. It’s probably better cos you actually need to think about what you’re building rather than just throwing services left and right at the problems.

I’m also not suggesting optimising and being careful is an argument for cloud. I’m saying that ruling out cloud is stupid. You can absolute have a Low cost solution perform very well on a cloud provider. The OP seems to think it’s not possible.