[agumonkey]

It also depends on a coherent structure in HTML websites.

Domains running websites which are more like javascript frontend modules shouldn't be scraped at all, it screams for a public API.

[uptown]

"it screams for a public API"

But many content owners would never provide their data in this format even if doing-so would be trivial.

[CMCDragonkai]

Try using https://snapsearch.io/ It is designed for JS sites.

[jdavis703]

These single page sites do have a public, albeit, undocumented API. If you analyze the network requests via the dev tools in your browser you'll have an XML/JSON data source that is probably structured better than the markup.

[agumonkey]

Of course, I should have thought about it that way.