|
|
|
|
|
|
by primaryobjects
4767 days ago
|
|
|
WhaleEye SEO Spider - http://www.whalei.com I created the same thing a year ago. I underestimated the CPU load and number of pages on many sites (especially e-commerce sites). It doesn't take many crawls to quickly fill a db. |
|
|