[jsnell]

Why do you think TikTok is ignoring robots.txt? This site is disallowing a lot of crawlers via User-Agent matching, but not ByteSpider.

https://www.nerdcrawler.com/robots.txt

The domain serving the images is allowing everything:

https://res.cloudinary.com/robots.txt

[chptung]

So I decided to google around and I think TikTok does ignore robots.txt. A few posts from around:

- https://www.reddit.com/r/flask/comments/161fqml/what_to_do_a...

- https://www.reddit.com/r/sysadmin/comments/1ahhzdg/has_anyon...

[catoc]

I just assumed that would have been the first thing they tried…