|
|
|
|
|
|
by AznHisoka
4517 days ago
|
|
|
Rand replied that they don't use AWS for crawling. I would imagine the main bottleneck, by far is the I/O with reading/writing to the database clusters and search index. Crawling is relatively cheap, it's storing that data that's hard. |
|
|