|
|
|
|
|
|
by alt_f4
2433 days ago
|
|
|
First off, robots.txt is optional. It's neither a technical nor a legal limitation at this point. Second, OP's argument suggests a UI option that gives or removes user consent from all robots in general. Unless they plan to word it: "allow robots that we like that are good for us. but disallow other robots", I don't think it's okay to discriminate by either allowing/banning a particular robot, as that is not what the user agreed to. |
|
|