[oldjokes]

I have worked at two startups now where we made the fatal mistake of being profitable. If you make this mistake then the investors will swoop in and demand you spend more on marketing and AWS infrastructure, because we're scaling up to 5 billion users of course.

Of course we started spending all the money on new people and AWS, and soon there was no money.

At one point we were dumping like $15K a month on AWS for a dozen unnecessary over-engineered toys that nobody was using. This is the real cost of AWS.

I'd love to see Amazon's data on money invested vs actual user traffic for small startups, that's got to be some of the most interesting and valuable data on earth. Forget companies, I'll bet Jeff is sitting around predicting when entire industries rise and fall weeks before anyone else based just on this data.

[matwood]

AWS or not, investors invest for the sole purpose of spending money with the hope of finding a unicorn. If an investor just wants to sit on their money, they can do that without giving it to a company.

[icxa]

Then those investors are stupid. Hope is not a strategy. But more likely is this take is wrong. Also sounds to me like GP maybe missed another consideration: architecting your application in such a manner that scaling is straightforward (not easy, straightforward, there is a difference).

Using AWS as an example, at one of the businesses I worked at, we used Kinesis as an event bus. One shard handles 2MB/sec of output. This worked pretty well for thousands of messages a second, we even got up to the 100's of thousands by compressing the payload of the event message. After that, you can employ any number of strategies that work easily, such as shading and adding additional streams, and use a lambda to pipe the output of one stream to another. It scaled up to millions of messages by essentially pushing buttons in the AWS Console.

Take a look at your architecture. More than likely, outside of a FB/Google/Netflix traffic scenario there is probably an easier and more straightforward architecture you can use that scales to your realistic use cases. Worry about the billions when you get there, which you yourself probably wont at that point because you would have exited, or moved into a higher role most likely by now.

[tyingq]

On the other side of the spectrum, AWS's extensive cost report metrics via tagging are great for big companies.

I can now show exactly which departments and dev teams are driving all the costs, and on what (CPU, storage, network). In a way that I never could for on-prem stuff.

[joncrane]

...sure, as long as they tag their resources properly.

The closest I got to an org that did this well was a big company that ran Cloud Custodian in all their AWS accounts and if you launched an EC2 instance, it would terminate it immediately with extreme prejudice if it didn't have values for three required tags, one to identify the "owner" individually and two for accounting purposes.

The only problem with that is there's no mechanism to make sure that the values of the cost centers values were correct. There was a bit of a scandal when one group (who presumably just copied and pasted a bunch of CloudFormation from another group's repo) was running 5 figures a months of infrastructure under the other group's billing codes.

ALSO, as many have said, bandwidth is a big part of the cost, and at this time it's nearly impossible to do showback/chargeback on bandwidth. There may be a way to do it using Flow Logs by correlating IP addresses to instances and using those tags, but I've never heard of someone doing this successfully.

[tyingq]

Egress charges leverage tags now. You can get down to good detail. Here's an image showing it will use tags if you set them: https://blog.cloudability.com/wp-content/uploads/2017/02/dat...

In this case, a service tag, set in some cases, not in others.

[alien_]

A better way than tagging is to give each team an AWS account to maintain and pay from their own budget.

[pm90]

Then you have to manage a million different AWS accounts. Each of them may be set up differently.

[scarface74]

That’s what CloudFormation and Organizations are for....

[tyingq]

You probably still want tags to break down costs by dev/test/prod, subsystem, etc. Or tags to aggregate them by department, customer, etc.

[altmind]

AWS cost reporting is far from great. Its hard to learn current(daily) charges; RI is completely hidden and only visible in the final bill, blended; there's no way to limit the spending and the detailed reports are in csv, not user-readable.

[tyingq]

Yes, there aren't great AWS provided tools, but the data is there. We happen to use Cloudability, though I'm sure there are other good tools, maybe even free ones.

[dillonmckay]

Can you elaborate on that or do you have any best-practices for tagging and correlating costs?

[tyingq]

We use a commercial tool. But, the most important tags are environment (dev, test, prod), application name, app version, and owning team.

For some apps, perhaps a "component" or "service" tag would also be important.

[geggam]

Only 15k a month ?

Startup I was at burned 400k/month average sometimes when buying RI we spent 800k.

Had a couple engineers route the database connections over the public load balancers for a month, that cost 20k in network alone.

AWS is not cheap at scale, period.

[Dunedan]

A startup being able to spend 400k/month on AWS should also be able to employ somebody to keep an eye on the AWS cost and look for possible optimizations. If that's not the case I wouldn't blame AWS for the spent money.

[geggam]

No blame was on AWS. The TAMs tried to help fix things but no one was listening. Many times I was the only person meeting the 2-3 TAMs we had on the account.

I have many stories about this startup. The idea is great. If the startup succeeds it will be in spite of how its being managed technically

[aasasd]

Could you please slip me in the bills at your next place? I'm currently spending less that 500 bucks a month.

[meritt]

What was the monthly revenue for this startup? Those are absolutely insane numbers.

[geggam]

Not sure what the monthly revenue was but I do know there were several multimillion dollar deals every 6 months or so.

Problems come from many areas... like the fact it was in the data management space and the 1500 or so i3's running Cassandra plus the hybrid cloud approach of front end in AWS talking to APIs in GKE which talked to backends in AWS because cool technologies you know.

No architecting was done. Build this and put it in the cloud. The team from RU didn't even know about autoscaling groups and when I tried to bring them in I ran into resistance.

I have many examples of "opportunities to optimize" from this place, needles to say my stress level is much lower not being there.

[mahmoudimus]

If you're taking money from investors and you're interested in being profitable, then you're going to sacrifice growth for the sake of profitability:

  - Let's not hire for this idea because it eats into our burn
  - Hmm, let's hold off on launching this feature until we have more data

Investment $ means taking risks (within reason) to maximize shareholder value.

If the company you worked for was profitable, then they could've structured a leveraged debt payoff to the investors to get them off the cap-table. Unless the company took so much money that the investors owned 60%+ and they unanimously do not agree about being profitable, then this is something that can be passed as a board resolution.

It sounds like the founders at your company were just inexperienced.

[dillonmckay]

What is the rule of thumb for cloud-spending costs of production versus development/CI?

Somebody mentioned 1:3.

[mgkimsal]

> If you make this mistake then the investors will swoop in and demand you spend more on marketing and AWS infrastructure

Would there be any "conflict of interest" issues is some of the same people requiring AWS spending were also Amazon shareholders?

[oldjokes]

Dude. A bunch of billionaires just got away with completely tanking the economy and getting bailed out for it. Zero people ever even talked about the possibility of anyone going to jail. Everyone got their bonuses. Nobody suffered any consequences. Literally nobody.

If you think anyone cares about any conflict of interest among the investing class you're beyond naive, you're just delusional.

[grogenaut]

How is Amazon supposed to know actual user numbers? That's app specific.

[biesnecker]

You can't get exact numbers, but for consumer-facing products you can estimate it via unique IPs hitting the load balancers, etc.

[grogenaut]

That would require amazon gathering customer information and possibly GDPR data on the usage of all of it's customers, which is a thing amazon doesn't do, and you don't want your cloud providers doing.

[fouc]

They probably meant page views and such.

[pbhjpbhj]

5 billion users? That's substantially more than Facebook .. was that an error?

[archgoon]

Most likely that was an exaggeration of the demands made by the investors.

[manigandham]

That was a joke.

[atmosx]

If you are profitable, why accept investor money anyway?

[cheerlessbog]

Grow faster?

[atmosx]

Well then, be careful next time.