|
|
|
|
|
|
by judge2020
846 days ago
|
|
|
Legality isn't the question here. If you want to speak to the legality, anyone circumventing a robots txt that explicitly has your bot's user-agent and 'disallow: *' is unauthorized access (I imagine it's more nuanced for 'user-agent: *'). No website is required to allow anyone to visit and can discriminate against any client or software any way they want. |
|
|
The question was literally,
> What are the legitimate (i.e. legal) use cases for a product such as this?