Hacker News new | ask | show | jobs
by mkl 1179 days ago
Do you mean JavaScript? I have never run into content hidden by Java, but many pages load content dynamically using JavaScript.

I have found it's quite easy to snoop on those JavaScript API requests using the Network tab of Chrome Devtools, then copy the network request as a curl command for bash scripts or as JavaScript for browser extensions.
3 comments
tomashubelbauer 1179 days ago
> I have never run into content hidden by Java

Tongue in cheek: You'd never know - servers running Java code generating HTML pages have probably conditionally not-rendered many pieces of HTML that you've never come across in your browsing :)

link
ghqst 1179 days ago
Yeah, you can sometimes find the API or find data sent in JavaScript but not in prerendered HTML, which can save you the pain of headless scraping.
link
juujian 1179 days ago
I do mean JavaScript. Not sure how many times I have made that mistake... And great advice, that sounds like a neat approach.
link