[samlambert]

This is definitely a lesson in the importance of indexes in general. We are well aware of the potential pitfalls with our current pricing. I’m happy to say we are nearly done modeling different metering rates for the product which would mean significantly lower bills for our users and avoid issues like this.

It’s core to our mission that our product’s pricing is accessible and friendly to small teams. Part of being in beta was us wanting to figure out the best pricing based on usage patterns. That work is nearly done. As the post mentions we’ve credited back the amount.

[anglinb]

Thanks Sam! As mentioned in the post, the PlantScale team was quick to credit our account for the overages and help us figure out what was going on. I'm personally super bullish on PlanetScale!

With any new product there will be tradeoffs and rough edges but the positives, like easy migrations and database branches have definitely outweighed any difficulties.

[brasetvik]

Kudos for being open about your mistakes.

Could you share a little bit about what your thought process was in general when picking a database technology?

You call out "easy migrations and database branches" outweighing other quirks, so some pros and cons weighing must've happened :)

Is it easy, for example, to test things in your dev environment with realistic amounts of data, and to get an understanding of how the queries will execute, etc? These seem somewhat basic, and would've probably caught this (also kinda basic, sorry :) problem early. (As in discovering "why is this query that should be a few ms with an index lookup taking so long?" early on)

[anglinb]

Thanks! The decision making process was pretty unsophisticated tbh. Basically I spent the last couple years working at GitHub as a security engineer and had been pretty comfortable with MySQL so wanted to stick with that. I had heard from our database team how annoying migrations were and I had previously locked a postgres database multiple times in production trying to deploy a migration, so MySQL + safe migrations + some of the best engineers I've worked with pouring all their time into PlanetScale, made a ton of sense. So basically a combination of proven underlying tech + believing in the team.

The migration workflow is really cool, basically when we create a PR we branch our production database and apply any migrations that are included in that PR and then that branch is used in our per-PR review environment. (Just Heroku Review Apps), then when we merge the PR, we also merge the deploy request in PlanetScale. Database branching is a super powerful concept once you've leaned into it.

We don't really do any sort of load testing in a dev environment. We have one customer who is also a co-founder of our company so we just deploy whatever changes we're unsure about (after automated testing) to his application and see what happens. If anything looks off in Grafana we'll make take a look but it's usually "good enough" or "totally broken", very rarely do we take time to make something 25% better if it already works. The time to fix vs speed of shipping features tradeoff doesn't make sense for us.

In this specific case, the query was taking place in a background job so 10ms - 500ms didn't really matter to us so we didn't really measure the timing, if we had we may have noticed it was slow but kinda a testament to PlanetScale that we didn't even notice ;)

[brasetvik]

Thanks for the background :)

Lock behaviour of migrations are indeed very important to be aware of.

Braintree (which are heavy Postgres users) have a pretty good post on that - https://medium.com/paypal-tech/postgresql-at-scale-database-...

[samlambert]

We are so glad to have you as customers and I can't wait to partner as you grow to mega scale.

[andybak]

Glad you came here to say this because my takeaway was "What a terrifying pricing model. That would keep me up at night"

[Nextgrid]

Would it be possible for you to add a spending cap? The user should be able to tell your system "here's how much I want to spend max" and if they exceed that they start getting errors or reduced performance.

[samlambert]

Yes absolutely. We have daily and monthly budgets on the roadmap to make sure nobody gets a surprise bill in the future. This and more tools to make sure you are running your database optimally.