[evanelias]

To echo and add to rwultsch's sibling comment:

* Facebook had extremely frequent schema changes, and powerful declarative schema management automation to support this

* The TAO (or more correctly "UDB") use-case supported using many separate tables, not one giant generic key/value table as people seem to assume

* The non-UDB MySQL use-cases at Facebook, in combination, are still larger than the vast, vast majority of all other companies' databases. These non-UDB databases use a wide range of MySQL's functionality. The frequent claims that "Facebook used MySQL just as a dumb K/V store" are absolutely incorrect and have never been correct.

[throwdbaaway]

Thanks for the insight! So how frequent does schema change happen to UDB?

[evanelias]

Sorry, I don't recall the exact frequency, I left FB in 2015.

[throwdbaaway]

Then I guess my point about UDB doesn't require schema change will still stand then? Looking at https://github.com/facebookarchive/linkbench, the table structure looks pretty much "final". And judging by https://dom.as/2015/07/30/on-order-by-optimization/, the access pattern is pretty much fixed as well.

[evanelias]

UDB had hundreds of tables per shard, and although there are a few common patterns, they did not all have an exactly identical structure. You have two former FB database engineers in this thread (myself + rwultsch) telling you your statement is incorrect. Nothing in Domas's post discusses lack of schema changes or tables being identical.

Linkbench is unmaintained and does not attempt to mirror the entirety of UDB, just its access patterns: point lookups by PK, and range scans over a secondary index. A fixed access pattern is not the same thing as having no schema changes.

Even putting column changes aside, the entirety of UDB was migrated from InnoDB to MyRocks in 2017, which is essentially a schema change across every single UDB table in every single UDB shard.

And besides, as I mentioned already, the non-UDB MySQL use-cases at Facebook are larger than the vast majority of companies' databases -- larger than the next-largest US social network, even. The non-UDB tiers had dozens of schema changes every single day.

As rwultsch correctly mentioned, Facebook's extreme agility with schema changes is directly what inspired me to create https://www.skeema.io, an open source project offering declarative schema change management. It's used by GitHub, Twilio, and a number of other well-known companies.

Please stop making incorrect statements based on things you have no direct experience with.

[throwdbaaway]

From my point of view, neither of you have sufficiently answer the question -- does UDB go through schema change that would require rewriting the table via pt-osc? If so, at what frequency?

Until then, sorry, we will keep advocating people to stay away from MySQL (and thus indirectly, skeema), because "long wait and potential incident from schema migration" is just not something that should come up during a sprint planning.

> Even putting column changes aside, the entirety of UDB was migrated from InnoDB to MyRocks in 2017, which is essentially a schema change across every single UDB table in every single UDB shard.

I read about that, and it is definitely an impressive feat, but still, that doesn't answer the question, at all? That's just relying on MySQL native replication that works across different engines.

[evanelias]

> does UDB go through schema change

Yes!

> that would require rewriting the table via pt-osc

Facebook does not use pt-osc; they use fb-osc, originally written in PHP and released in 2010 [1] and later ported to Python in 2017 [2]. The concepts are similar to pt-osc re: core use of triggers, but the fine print has some important differences about how the new table structure is specified, when changes are applied, and how changes are made on replicas.

Anyway, fb-osc was used on UDB, the answer is emphatically yes.

btw I am using the past tense here because I haven't kept up with FB mysql stuff the past few years. I don't even know if UDB is still on mysql at all; it's irrelevant to the discussion because the key point here is that schema changes emphatically did occur on UDB for many years, and your original statement regarding TAO and schema changes was demonstrably false, full stop.

> If so, at what frequency?

As I already said, I do not recall! Why would I remember the exact frequency of a completely and seamlessly automated process at a company I left 6 years ago?

UDB didn't require fb-osc changes nearly as often as non-UDB, if that's what you're asking, by nature of it serializing most (not all!) object fields down to a single column. But there were definitely still cases where actual schema changes were necessary on UDB tables, as I'll say yet again, the schema was not completely uniform across all UDB tables.

What's with this obsession on the frequency, anyway? Why does this matter? Your original statement was "the schemas are pretty much static, and they don't feel the schema migration pain", and this statement was wrong. Stop moving the goalpost.

> Until then, sorry, we will keep advocating people to stay away from MySQL (and thus indirectly, skeema), because "long wait and potential incident from schema migration" is just not something that should come up during a sprint planning.

Until when? Who is "we"? You aren't making sense. First your argument was that Facebook supposedly doesn't make schema changes at all, and now you're seemingly pivoting to bashing MySQL for needing external OSC tools, even though your original comment directly acknowledged that PG has cases where lack of these tools is a major problem?

> but still, that doesn't answer the question, at all? That's just relying on MySQL native replication that works across different engines.

What's relying on native replication? Changing a table's storage engine inherently requires rewriting the entire table.

[1] https://www.facebook.com/notes/mysql-at-facebook/online-sche...

[2] https://engineering.fb.com/2017/05/05/production-engineering...