Before the scaling problems were hit they were hosting in 4 very powerful machines with loads of RAM and all-SSD storage. I don't see any reasonable world in which those machines aren't enough to power 30,000 users.
The issue was the media storage. Mastodon stores lots of small files in deeply nested structures, so filesystems, especially networked filesystems, are very badly suited. Not to mention the disks themselves having issues.
The article quotes blocking requests for 10 to 20 seconds, this causes everything to go slower.
> Mastodon stores lots of small files in deeply nested structures
The deep nesting and overall layout of the cache drives me nuts. The storage layout is designed for the state of filesystems ca. late 90's, pre reiser or ext3, with the major deficiency of pretty much demanding an unnecessarily expensive object storage setup.
The nature of most of the space on an instance like this going to cache remote content is also that it ought to not try to make caching decisions in the app, but leave the actual caching to a proper cache, which would've been trivial if the paths of the cached objects was mapped to the media-proxy URLs it falls back to if the file has not already been downloaded. Instead a bunch of effort goes into building a cache setup that people often end up putting on expensive object storage when it's data that shouldn't even need redundancy.
Eh, we served way bigger things, with millions of small files via NFS just fine. Out of non-SSD storage too. Their issue, as written in article, was SSDs failing.
It was migration from one clustered filesystem to another and the temporary step was just Apache equivalent of try_files and NFS mount aside the new FS while the migration was running in the background.
If anything filesystems are better with some nesting vs "one big dir"...
I believe it was I/O bound, on old, failing SSDs. Even after migrating object storage out to DigitalOcean's S3-equivalent, Postgres couldn't keep up on the terrible disks.