[ivanovb]

I was working recently on a similar project and I knocked my head in the GDPR walls; the most annoying being TechCrunch - you don't just get a modal that you could bypass, they send you to another site.

Did you find a way to deal with this hindrance ?

[eafer]

The core of my code is a line by line translation of the Firefox version. I know what it does, but not the exact motivation for everything, so it has many hidden tricks I never noticed. I'm not in Europe and I never tested this, but it's possible that it does remove some of the modals, as long as the actual content is on the page.

It won't do anything for the TechCrunch case you describe, because it only fetches the one webpage you point it to (and any redirections).

[danpeddle]

I took a very quick look at the source code, and seems you’re using the curl default options for things like the user agent. Please correct me if wrong!

Did you try pretending to be a search engine crawler, for an idea..?

[eafer]

I didn't; I guess I could let the user pick an agent. You think that would help with GDPR?

So far, I've never run into this problem myself. If I did, I think I would use tor.