Hacker News new | ask | show | jobs
by jxramos 1224 days ago
Glad to see this text

""" What happens if my scraping fails? Not to worry! We will make every effort to determine the cause of the problem and assist you in resolving any issues with your scraper.

Additionally, please note that unsuccessful scrapings will not be included in your monthly quota. """

I'm curious the feedback mechanism for failed scrapes. Is there any validation configuration or an email notification I can configure in the event the target changes their page layout or DOM or whatever happens to cause interference.

1 comments

If the target page or DOM changes, it's not considered a failed scraping, just the data you wanted will return empty.

But you gave me the idea to send an alert or email notification if a scraper stops returning data or content changes.

Thanks

totally, that's going to be a huge piece of functionality that will add a ton of value to the user. It's what I always worry about in maintaining my scrapers, updating xpaths, all the rigmarole parsing and what not. Stable pages are much appreciated but we all know web designers are gonna design. Thanks for making this product, being in a post https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn world is gonna make for some great adventures I think. Keep up the good work.