[bbor]

Do these services respect norobot manifests? Isn't this all kinda... illegal...? Or at least non-consensual?

[basilgohar]

robots.txt isn't legally binding. I am interested to know if and how services even interact with it. It's more like a clue on when the interesting content for scrapers is on your site. This is how I imagine it goes:

"Hey, don't scrape the data here."

"You know what? I'm scrape it even harder!"

[bbor]

Soooo nonconsensual.

Maybe bluesky is right… are we the baddies?

[tonyhart7]

it is legally binding if your company based on SV (only California implement this law) and they can prove it