|
|
|
|
|
|
by fullstacking
29 days ago
|
|
|
heres the rub, whos to say in the future or even now that google is really respecting the robot.txt in the first place. I know for a fact that my sites are still getting hit by AI bots but I'm "blocking" them via cloudflare. Way too many IP's originating from data center ISP. Just saying respecting the robot.txt maybe just PR lip service. |
|
|