Hacker News new | ask | show | jobs
by caydenm 886 days ago
OP here, I was trying to say that these pages were behind an authwall and loading with userids from a specific user but without any of their cookies to support that auth.

This led us to believe this page was MitM rather than crawled directly (as they would not be able to impersonate the user)

1 comments

That's how I read it also. If the ids you're referring to were in the URL, it's almost certainly URL Filtering. The URLs are fed to the crawler via MITM, so you were basically right.