|
|
|
|
|
by klapinat0r
5128 days ago
|
|
> They saw the referrals from our sites, then saw that we were displaying showtimes without being licensed to do so (how they knew this I'm not sure) The referrals are likely the headers you send when scraping. i.e. Referer: <your newspaper>.tld. Depending on whether you actively set the User-Agent header, that might also have contributed to them catching on (be it omitted User-Agent, "urllib2", "<newspaper> Bot 1.0 +<newspaper>.tld; don't sue us", and so forth). If you run a content provider, and try to protect your content/pageviews/API, the lack of either of these headers is also worth looking out for. |
|