[benoau]

> how would serverless help with spidering and indexing?

Because these are tasks that lend themselves to massive concurrency and serverless makes that instantly possible. If you were going to put this on your own hardware that will be a pretty big project of its own.

How many months of dev time can you budget for rolling your own indexing/spidering infrastructure and getting that just right? How does that affect your launch date?