[__loam]

You're taking someone else's labor and profiting off it, without any credit or compensation. To add insult to injury, the person you're scraping pays money to support your traffic. It's a one sided transaction.

[9dev]

You can’t generalise that. Maybe I crawl to provide an annotated preview of their website, to make users of my application more likely to click the link and visit it? There are lots of ways in which crawling benefits everyone, it just requires some mutual respect.

[deisteve]

We don't live in the 90s anymore. The bandwidth CPU cost is moot unless you are spinning up thousands of GPUs to render an HTML page.

Also the claims of someone else's labor and profiteering is exactly what Google does

[__loam]

What Google does is mutually beneficial. Stealing content to reheat and serve in an LLM is very different.

[deisteve]

Google that creates monopoly on top of the scraped data and uses third party content to do so?

LLM just places that ability at the ends of the user (local LLM strictly speaking)

[__loam]

No it doesn't. The end state of a Google search is a human interacting directly with a site.