Hacker News new | ask | show | jobs
by ianhawes 4748 days ago
Something to consider is that the trend the past year has been to use headless browsers over BeautifulSoup, cURL, etc.. because headless browsers are harder to detect by anti-scraping systems and can interpret JavaScript.
1 comments

That's what the OP is about ;-). But BeautifulSoup isn't a way to retrieve a web page, it's a way to parse HTML. You can get the page with a headless browser, and then transfer the DOM into a BeautifulSoup tree to do your scraping.