Hacker News new | ask | show | jobs
by unreal37 2943 days ago
You had a paid API, and people wanted the information for free....

Not unexpected I guess.

3 comments

After searching "algolia" mentioned below, I figured out the misunderstanding. The company was paying somebody else per search made on their web site. So every time a scraper called the website's search function, it cost the website money.
you've misunderstood: the site used something like algolia (which the site paid for) to index. the scrapers were hitting that service (which was costing the site) rather than parsing the xml (which already had everything).
The paid API was just to handle the search of the company database (couldn't change that for political reasons).

They weren't getting any information that they couldn't get through the XML.