[kyleleelarson]

I am working on a somewhat similar project, for searching items 1 and 1a in 10-k annual reports, that I am hoping to release in the near future. I would be interested to hear what lessons you end up learning about scaling up to handle the interest you got from HN.

[anonyonoor]

Definitely limit the more disk heavy features, or spend more time (and money) on infastructure. I was running the whole site on an 8vCPU 24GB RAM VM, and it almost immediately crashed due to the high disk reads.

This is likely due to the fact that the database is huge, and providing that data on demand is very resource intensive- especially when there are forty different people sending many requests a second.

If you don't want to compromise on data though, look into spending a little bit more time/money on infrastructure. I wish I had deployed the project on Kubernetes, instead of what I ended up doing.