|
|
|
|
|
|
by bambax
1153 days ago
|
|
|
Can you crawl websites on a private network not accessible from the outside (by downloading a tool that would run inside the network for example)? Your front page says "any url public or private" but it doesn't really answer that specific question. I think a tool that would work inside corporate networks, and maybe be runnable on a simple notebook without requiring admin privileges on Windows, would have a significant market. |
|
|
You are absolutely right about a local crawl offering. In fact it would cut down a lot of our server costs too -;) Will be looking into this at our next standup since it will require some refactoring.
There is however a massive trade-off in that crawling and screenshooting large >5000 pages on one cpu will take a really long time to do well because there is a complex screenshot stitching process, and that process will need an ALWAYS ON CPU.
What we are offering as of this month is enterprise custom cloud installs & licensing. That way 5k site will get done in 1-2 hours ( or faster ) instead of days!