|
|
|
|
|
|
by netdevnet
617 days ago
|
|
|
Some of the scrapers used by big companies do identify themselves as bots by using unique user agents. Of course, it does not mean that they don't have other bots running around without the bot user agent name. Whether those days are over or not will greatly depend on the outcome of the ongoing New York Times vs OpenAI lawsuit. If OpenAI wins, then it pretty much green lights all the other scrappers to feast upon the web |
|
|