Hacker News new | ask | show | jobs
by dogma1138 224 days ago
Do they actually do anything to circumvent paywalls or do websites just whitelist their crawlers?
1 comments
walletdrainer 223 days ago
Websites don’t whitelist their crawlers, they maintain custom bypasses for a wide variety of websites.

If the websites were inclined to whitelist these crawlers, they’d also whitelist archive.org which is actually easy to whitelist. Archive.is is not

link