[II2II]

> But now I'm really reluctant to give more stuff to the free web. Because the fact that it gets scraped and added to a pile of training data to later be monetized really rubs me the wrong way.

The thing is, people were scraping and monetizing other people's websites long before the current LLM fad. The difference is the magnitude of the problem.

> And I can see why maintainers of sites like this, or other free but incredibly useful resources might start to get irate at that.

Those who object to the scraping fall into several camps, but the biggest complaint I am hearing is that it increases both maintenance costs and time. In other words: it sucks when people are using your work in a manner that you find offensive, but it goes beyond that by doing actual harm.

[antisthenes]

> The thing is, people were scraping and monetizing other people's websites long before the current LLM fad. The difference is the magnitude of the problem.

People doing it with a couple of machines and residential proxies versus Anthropic doing it with 2 data centers worth of machines.

Scale matters.

[joshmarinacci]

A change in quantity can become a change in quality.

[rjtavares]

I think the toxicology proverb applies here perfectly: The dose makes the poison.

[dotancohen]

I believe that it was Stalin who phrased it most eloquently: Quantity has a quality all its own.

[danlitt]

Not Stalin: https://klangable.com/blog/quantity-has-a-quality-all-its-ow...

[dotancohen]

Thank you!

[TeMPOraL]

> The thing is, people were scraping and monetizing other people's websites long before the current LLM fad. The difference is the magnitude of the problem.

> In other words: it sucks when people are using your work in a manner that you find offensive

That to me still reeks of Dog in the Manger mentality. If you publish something for the world to use, you should neither care nor even track, much less discriminate by (or suddenly seek compensation for) who is using it.

[Cthulhu_]

Agreed, this is what you agreed to (implicitly or explicitly) when uploading stuff to the internet. In fact, open source embraces this (hence the 'open').

But people are free to not publish things or post things online with a more restrictive license. Not that a license stops things from being indexed.

[II2II]

> Agreed, this is what you agreed to (implicitly or explicitly) when uploading stuff to the internet.

A more pragmatic approach is to acknowledge that concerning one's self with how something is used once it has been released is an emotional drain. It is a bit much to suggest that someone agreed to something, even if that agreement is implicit, just because they released it.

> But people are free to not publish things or post things online with a more restrictive license.

Licences are meaningless unless you have the ability to enforce them (e.g. to sue). That's why so many companies are willing to ignore the terms of open source licenses. It's also why the attempts of enforcement that we do hear about are usually backed by a third party, rather than being done by the software developer themselves. Simply put, the individual developer (or even small project) trying to make a contribution to the community would be better served by not publishing (instead of using a restrictive license) if they are concerned about how their work is used.

I don't even know if there is a good way to resolve the problem. Consider something like a DMCA Takedown notice. It removes the administrative and legal overhead to copyright infringement, yet it is also easy to abuse. For example: businesses have weaponized it by using it against individuals. Perhaps my cynicism is taking over here, but I suspect any easily accessible mechanism for enforcement would be similarly abused.

[TeMPOraL]

> But people are free to not publish things or post things online with a more restrictive license. Not that a license stops things from being indexed.

Right. In fact, people are also free to publish things with licenses that condition access on compensating the author/publisher, and they have both social and legal backing to enforce it. This is called "proprietary", and it's not a wrong choice - in fact outside of software, it's the default choice.

The problem is when people publish "free" and "open" as a marketing tactic, where in fact they really want to control and charge for access (whether dollars or karma or credit). That is just plain dishonesty.

[amiga386]

This is absolutely fine, until some shitheel decides they're going to continually scrape your website, thousands of requests per second, and knock it offline constantly, so they don't have to cache even a single thing, or write better scrapers. They'll just suck up your bandwidth for it. After all, it's you that's paying, not them.

Search engines were absolutely fine. They were respectful of the sources they scraped. There's now a bunch of money-obsessed shitheels doing whatever they can, as blatantly and as lazily as they can, because they believe they'll become rich. Every one of these fuckers needs to be dead or in jail before the world becomes whole again.

If nobody takes action against these fuckers, everything you care about will just nope out of existence, and these fuckers will be all you have left.

[TeMPOraL]

Major AI companies aren't indiscriminately scrapping any more than search engines were (which themselves caused some problems in the past too, until that got sorted out). Your gripe is with SEO people and content marketers.

I'd be more sympathetic if it was really about this, though. However, the majority of the complaints seems to come from people personally offended by the possibility of their content, which they claimed was published for everyone to benefit, ending up as training data, and thus actually benefiting humanity, at scale far beyond the original publication ever would. Which is the Dog in the Manger attitude I point out.

[amiga386]

The problem is the rat race for the pot of gold at the end of the rainbow.

A similar thing happened when crypto was in ascendence. Web pages were getting crypto-miners injected into them. Celebrities were shilling NFTs. Everyone and their dog was on the make. The thought of riches broke the minds of millions.

The same is happening with AI. Whether it's the "major AI companies" or millions of self-interested also-rans with fewer moral scruples scraping, the problem still exists. The problem will continue to exist until you can't conceiveably make money by scraping like a bastard. If everybody identified themselves up front and respected robots.txt, there would not be a problem. But they don't, and they don't, and they pummel websites for no fucking reason, and they don't care, and they won't stop.

Websites get hammered by millions of unique IP addresses from residential ISPs which happen to belong to botnets, none of them identifying themselves as a bot user agent, all just pretending to be some slightly out-of-date version of Chrome.

And it's not like the "major AI companies" hands are clean. Who bought up all the RAM and compute power? Who's building datacenters in the third-world parts of the USA and powering 24/7 them with diesel generators, or sucking up most of the power generator, and of course using up all the drinking water to cool their machines?

Everything in humanity and nature is there for the AI bros to use and dispose of, provided they come out on top. This is the attitude that needs to be taken down.

[TeMPOraL]

You may be right about the scale of also-ran operations, even though I disagree with the core comparison. Unlike crypto coins, which require superlinearly growing energy waste just to sustain their basic guarantees, and were created to solve "problems" that aren't problems and don't need solving (hint: trust is a feature, not a bug), AI actually works. It delivers real value for cheap. The growth isn't artificial, it's a product that doesn't even need much marketing[0] - it exploded organically since ChatGPT, since it's obviously that immediately useful for approximately everyone in some aspects of their work or life.

One nit though:

> And it's not like the "major AI companies" hands are clean. Who bought up all the RAM and compute power? Who's building datacenters in the third-world parts of the USA and powering 24/7 them with diesel generators, or sucking up most of the power generator, and of course using up all the drinking water to cool their machines?

They paid for it, much like everyone else.

> Everything in humanity and nature is there for the AI bros to use and dispose of, provided they come out on top. This is the attitude that needs to be taken down.

Here you're arguing against the basic market economy. They aren't using and disposing of anything they couldn't buy for that purpose like literally everyone else. There's no theft or trickery going on here. There's a boom, because AI is that useful, but it's still all normal resource allocation.

--

[0] - Of course the competing players invest tons in marketing to gain an edge against the other players.

[amiga386]

> They paid for it, much like everyone else.

That's really not how it works. Did you miss the part where the RAM vendors, had they known their rivals were also selling all their RAM to OpenAI, would not have sold what they sold? OpenAI, who did know this, and what it would mean for every other industry and device on the planet that needs RAM, chose to go ahead with it. Because all they care about are themselves, fuck everyone else I got mine

Markets are imperfect because of information asymmetry, aka the Cheating Bastard Problem.

After announcing the deal and the truth coming out, they could also have said "hey, we see now this crimps all other industry but ours. We'll make this right, we'll reduce our RAM order if our vendors would prefer that." But they didn't, because fuck everyone else I got mine

As the AI companies are so flush with cash, they could afford to offer a Common Crawl, so everyone who wants web resources for their models or whatever can get very fresh cached copies from a common crawler, which obeys robots.txt and crawls in a fair way so as not to overwhelm the source webservers. But fuck everyone else I got mine, so here come a million dodgy scrapers to bring down everyone's site.

And in terms of natural resources, one does not simply "pay" and get whatever they want. Any government worthy of the name puts its own residents needs above the lucrative offers made to them by amoral outsiders who couldn't care less if the residents lived or died.

If you're a consumer of products and you know the product was made by taking away vital resources from people elsewhere, then you're complicit in that action by continuing to buy the product.

[hack1312]

Your rhetoric is harsh but you’re not wrong.