[simplecto]

i hate to be that guy, but “it depends”

scrapy is still king for me (scrapy.org). there are even packages to use headless browsers for those awful javascript heavy sites

however, APIs and RSS are still in play, and that does not require a heavy scraper. I am building vertical industry portals, and many of my data rollups consume APIs and structured XML/RSS feeds from social and other sites.