Hacker News new | ask | show | jobs
by salimmadjd 3350 days ago
What if the content you want to scrape->API is behind a login gate? Is there an option of authentication?
2 comments

You can actually write an API endpoint that'll retrieve a state token (which includes the cookies). For one example, you can use view our Hacker News login endpoint at https://wrapapi.com/#/view/phsource/hackernews/login/latest

That endpoint will then emit a state token, which includes the session cookies. You can feed that state token into your next request and it'll authenticate you

Yes, you can get the content you want even if it's behind a login page! Expect to create 2 APIs, one for logging in and another for getting the content. An example is provided on their homepage: https://wrapapi.com/v2#/caseStudies/cj