|
|
|
|
|
|
by artur_makly
1152 days ago
|
|
|
today we crawl private sites that only use cookies as their auth method: https://support.visualsitemaps.com/en/articles/3673081-how-t... You are absolutely right about a local crawl offering. In fact it would cut down a lot of our server costs too -;) Will be looking into this at our next standup since it will require some refactoring. There is however a massive trade-off in that crawling and screenshooting large >5000 pages on one cpu will take a really long time to do well because there is a complex screenshot stitching process, and that process will need an ALWAYS ON CPU. What we are offering as of this month is enterprise custom cloud installs & licensing. That way 5k site will get done in 1-2 hours ( or faster ) instead of days! |
|
|