"proxy rotation" in the first line does not bode well for ethics.
Not every crawled site is "big tech" or even commercial.