[viraptor]

Wayback machine does look at robots.txt - https://help.archive.org/hc/en-us/articles/360004651732-Usin...

[foob]

They look at them, but they don't follow them strictly [1]. They make judgement calls on what they should do rather than treating robots.txt files as a legal contract.

[1] - https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...