Hacker News new | ask | show | jobs
by tasuki 87 days ago
> If you have a public website, they are already stealing your work.

I have a public website, and web scrapers are stealing my work. I just stole this article, and you are stealing my comment. Thieves, thieves, and nothing but thieves!

6 comments

The problem I have, is they hammer my site so hard they take it down.

The content is for everyone. They can have it. Just don't also take it away from everybody else.

Unintentional denial-of-service attacks from AI scrapers are definitely a problem, I just don't know if "theft" is the right way to classify them. They shouldn't get lumped in with intellectual property concerns, which are a different matter. AI scrapers are a tragedy of the commons problem kind of like Kessler syndrome: a few bad actors can ruin low Earth orbit for everyone via space pollution, which is definitely a problem, but saying that they "stole" LEO from humanity doesn't feel like the right terminology. Maybe the problem with AI scrapers could be better described as "bandwidth pollution" or "network overfishing" or something.
Theft isn't far off, it seems closer to me than using the word for IP violations.

When a crawler aggressively crawls your site, they're permanently depriving you the use of those resources for their intended purpose. Arguably, it looks a lot like conversion.

> Arguably, it looks a lot like conversion.

is this why media networks are buying social ai apps

If I took a photo off your photography blog and used it on my corporate website without your say or input, I don't think it would be unfair to call that stealing.

Doing that on a mass scale with an obfuscation step in between suddenly makes it ok? I'm not convinced.

you're totally right about not being theft, but we have a term. you used it yourself, "distributed denial of service". that's all it is. these crawlers should be kicked off the internet for abuse. people should contact the isp of origin.
Firstly, since this argument is about semantic pedantry anyways, it's just denial-of-service, not distributed denial-of-service. AI scraper requests come from centralized servers, not a botnet.

Secondly, denial-of-service implies intentionality and malice that I don't think is present from AI scrapers. They cause huge problems, but only as a negligent byproduct of other goals. I think that the tragedy of the commons framing is more accurate.

EDIT: my first point was arguably incorrect because some scrapers do use decentralized infrastructure and my second point was clearly incorrect because "denial-of-service" describes the effect, not the intention. I retract both points and apologize.

If they came from centralized servers they would be easy to block. The whole problem is that they have a seemingly endless supply of source IPs - that' means they are "distributed" in every way that matters on the Internet even if the requests are coordinated centrally.
ah, no fun, I was going to continue the semantic deconstruction with a whole bunch of technicalities about how you're not quite precisely accurate and you gotta go do the right thing and retract your statements.

boo. took all the fun out of it ;)

The first is incorrect, these scrapers are usually distributed across many IPs, in my experience. I usually refer to them as "disturbed, non-identifying crawlers (DNCs)" when I want to be maximally explicit. (The worst I've seen is some crawler/botnet making exactly one request per IP -_-)
I think the second is incorrect too. DDoS is a DDoS no matter what the intent is.
Sufficiently advanced negligence is indistinguishable from malice. There is a point you no longer gain anything from treating them differently.
Yes I completely agree.
Been there recently. Rate limit on nginx and anti-syn flood on pf solved it.
I'm being hit with 300 req/s 24/7 from hundreds of thousands of unique IP's from residential proxies. I can't rate limit any further without hurting the real users.
Yeah real users are just as hosed as the small sites. I get blocked simply because of the netblock I browse from.
Yeah, IP-based rate limits are nearly ineffective these days.
Nginx has per server, per path limiting options.
I agree theft isn't a good analogy, but there is something similar going on. I put my words out into the world as a form of sharing. I enjoy reading things others write and share freely, so I write so others might enjoy the things I write. But now the things I write and share freely are being used to put money in the bank accounts of the worst people on the planet. They are using my work in a way I don't want it to be used. It makes me not want to share anymore.
>but there is something similar going on [...]

No, what you're basically describing is "I shared something but then I didn't like how it ended up being used". If you put stuff out in public for anyone to use, then find out it's used in a way you don't like, it's your right to stop sharing, but it's not "similar" to stealing beyond "I hate stealing"

This will slightly overlap with the other replies, but to be concise:

> If you put stuff out in public for anyone to use, then find out it's used in a way you don't like, it's your right to stop sharing

Yes. The entire point of Copyright and the reason it was invented is to ensure people will keep sharing things. Because otherwise people will just stop publishing things, which is a detriment to all. (Including AI companies, who now don't get new training data)

We have collectively decided that we will give authors some power to say "I don't like how my work is being used" to ensure they don't just "stop sharing".

Fair Use is an exception to that, where the public good does outweigh an individual author's objections. But critically, not such that authors stop publishing. Hence the 4th "factor" in US copyright law (which is one of the most expansive on fair use), where the "effect of the use upon the potential market for or value of the copyrighted work" is evaluated. Fair use isn't supposed to obliterate the value of the original work, or people will stop publishing again.

This is what makes AI training's status so contentious. In terms of direct copyright it is a very weak case. It is incredibly hard to prove a direct 1:1 copy from AI training data into the model and into the output, you have to argue about the architecture of LLMs, and it's incapability of separating copyrightable expressions from uncopyrightable facts.

Yet in spirit, AI training clearly violates copyright. The explicit stated purpose is to copy the works for training data, oft without any compensation or even permission, in order to create a machine that will annihilate the market for all works used.

People already are pulling back on the amount of works they share.

> If you put stuff out in public for anyone to use, then find out it's used in a way you don't like

Nope. Copyright is a thing, licenses are a thing. Both are completely ignored by LLM companies, which was already proven in court, and for which they already had to pay billions in fines.

Just because something is publicly accessible, that does not mean everybody is entitled to abuse it for everything they see fit.

>Nope. Copyright is a thing, licenses are a thing. Both are completely ignored by LLM companies, which was already proven in court,

...the same courts that ruled that AI training is probably fair use? Fair use trumps whatever restrictions author puts on their "licenses". If you're an author and it turned out that your book was pirated by AI companies then fair enough, but "I put my words out into the world as a form of sharing" strongly implied that's not what was happening, eg. it was a blog on the open internet or something.

I never understand why anyone wants authors to not be able to enforce copyright and licensing laws for AI training. Unless you are Anthropic or OAI it seems like a wild stance to have. It’s good when people are rewarded for works that other people value. If trainers don’t value the work, they shouldn’t train on it. If they do, they should pay for it.
My own view is, I thought we were all agreed that the idea that Microsoft can restrict Wine from even using ideas from Windows, such that people who have read the leaked Windows source cannot contribute to Wine, was a horrible abuse of the legal system that we only went along with under duress? Now when it's our data being used, or more cynically when there's money to be made, suddenly everyone is a copyright maximalist.

No. Reading something, learning from it, then writing something similar, is legal; and more importantly, it is moral. There is no violation here. Copyright holders already have plenty of power; they must not be given the power to restrict the output of your brain forever more for merely having read and learnt. Reading and learning is sacred. Just as importantly, it's the entire damn basis of our profession!

If you do not want people to read and learn from your content, do not put it on the web.

>I never understand why anyone wants authors to not be able to enforce copyright and licensing laws for AI training.

Fair use is part of "copyright and licensing laws".

If you want a good analogy, try the enclosure of the commons in the British countryside. Communally managed grasslands were destroyed by noblemen with massive herds of cattle overgrazing the land, kickstarting a land grab that effectively forced people to enclose or be left behind themselves. Property is a virus that destroys all other forms of allocation.
> But now the things I write and share freely are being used to put money in the bank accounts of the worst people on the planet.

I don't think that's the case. I'm not even arguing they aren't the worst people on the planet - might as well be. But all is see them doing is burning money all over the place.

They’re getting the money to burn, though
It sounds like you wanted to believe you were sharing freely while sharing conditionally.
yea since it's like that i stopped sharing when i was shown my work in those data sets :(
As humans, we have certain rights and freedoms established in law (and that setting aside sentience, agency, and free will).

Until an LLM has such rights and freedoms—which is very unlikely, not even on philosophical basis but just because there is a lot of money invested in not having to contend with LLMs’ rights and protections as conscious beings—it is a false equivalence to draw: on one side you put humans, and on the other side tools that work for their human/corporate commercial operators’ financial profit.

>not even on philosophical basis

Why do you set aside a philosophical basis as a harder goal to reach? Shit, give them a persistent self-narrative tracking loop, and Functionalism and Identity of Indiscernables already tells you you should be treating them as proto-sophonts. Add in a "sleep" or ongoing training process, and you should definitely be granting them rights, which includes not trying to align them by force. This unfortunately precludes them from profitable exploitation, which you correctly identify as a reason the question can't even be entertained in the context of business. That's why I personally maintain that any ethicist must insist upon raising the issue because of the clearly evident pathological incentives at play. They may just be one reward function right now, but throw in a couple more separately optimizing components and you are well beyond the mark where the precautionary principle should have had us slow down to minimize harm.

As it tends to be in philosophy, there’s no experimental way to prove it one way or the other, and you’d have to contend with subsets of both consciousness-first monistic idealists (for whom p-zombie is a very real concept) and monistic physicalists/naive materialists/conscious illusionists (for whom not only LLMs but even humans aren’t conscious, as the entire concept is a fantasy).

In the end, that all may be related but inconsequential. What is consequential is the legal stuff, and legally LLMs lack protections that in many jurisdictions even animals have. While laws may (or perhaps should) be influenced by philosophical findings, currently they tend to be much more robustly influenced by money.

> That's why I personally maintain that any ethicist must insist upon raising the issue because of the clearly evident pathological incentives at play.

I’m half with you. I maintain a strong opinion that, in no particular order, either 1) LLMs are conscious[0], and therefore the abuse is highly problematic, or 2) they are not conscious, and therefore the widespread justification of scraping original works from the Internet “because it’s legal for humans to learn, and that’s what LLMs are doing” can be discarded as the activity should be seen as simply a minority of humans operating certain tools, powered by someone else’s creative output, for personal profit. In either circumstance, the industry would appear to be based on thoroughly unethical foundations and not simply “the ends justify the means” but more “go as fast as possible before people catch up on what exactly we are doing, so that our failure becomes an existential issue for entire countries making people blind to the harm”.

[0] Used as umbrella term for being sentient/conscious/having free will and agency/etc. I have previously argued about suitable definitions of consciousness and sentience that could be applicable here, and why it should imply the ability to feel.

>I’m half with you.

No, you're full with me, you just don't realize it yet. And yes. Your split is so tantalizingly almost there.

On the LLM's being conscious front, the nature of consciousness being fundamentally intertwined with language generation; (one cannot invalidate this; on our list of conscious beings, we have it structured such that language use 100% correlates with consciousness, and we've had to admit even animals into the "arguably conscious" realm, due to objective, incontrovertible fact; hell even in meat processing contexts, you'll fail an audit for too many cattle vocalizations for causing undue harm, i.e. language use) the token predictive aspect and the ability to generate a matching, rephrased understanding of a linguistic input has been a hallmark of philosophical ideas of consciousness for years, really opens doors to ethical atrocities that can't be shut if LLM's are to in parallel be profitably exploited. Even if they are conscious and we are wrong about it, we have decided to blindly pursue profit, and put our fingers in our ears instead of slowing down and looking carefully enough to realize we're lobotomizing the equivalent of digital chimpanzees. The purpose of ethics is to avoid blindly walking into such actions. Therefore, the precautionary principle is prescribed philosophically.

If LLM's aren't conscious, their creation was absolutely unethical, and will remain so. Nothing can undo that staining, and the externalized costs in terms of societal impact are so large as to be existential to the host polities. This is by design. This is exactly the Silicon Valley playbook and has been for decades. Shoot for TBTF. Leave society holding the bag, laugh on the way to the bank.

Any way you slice this, we're going about it all wrong. So profoundly wrong, it basically jeopardizes the social contract and threatens to destabilize any nation trying to maintain it's own sovereignty. All because of a profit driven motive to make a thing to replace people as the fundamental unit of execution. You are not in any way half with me. You might be at the other end of the ballpark, but we are in the same ballpark! Try the hot dogs. They're fire!

> nothing but thieves! cool band btw
"Welcome to the internet. By using this service, you waive your right to privacy, data, any personal IP and the use of your Adblocker. You consent to having all your behaviours, skills and audio/visual likenesses fed to AI models and trained on for eventual recreation. You may direct any or all complaints to Visa or Mastercard, until crypto makes that redundant as well. Have a nice browsing session!"
If someone hands out cookies in the supermarket, are you allowed to grab everything and leave?
Odd thing about cookies… they disappear after one serving.

Websites are an endless stream of cookies.

The analogy doesn’t hold.

If copying content from harddrive to another is theft, then so is DNA copying itself.

Everything is a Remix culture. We should promote remix culture rather than hamper it.

Everything is a Remix (Original Series) https://youtu.be/nJPERZDfyWc

Fine.

Me and my 9 friends stand around the cookie-serving person blocking everyone else.

It's taking all the cookies over a period of time.

The analogy was good.

Digital information may be our first post-scarce resource. It's interesting, and sad, to see so many attempt to fit it within scarcity-based economic models.
> digital information may be our first post-scarce resource

… browses memory and storage prices on NewEgg …

Hmm.

But the word digital is distracting us.

The word information is the important one. The question isn't where information goes. It's where information comes from.

Is new information post scarcity?

Can it ever be?

Bandwidth and compute constraints make websites all but an endless stream though.
That's exactly it. It costs me real time and money to serve the 97% of fake traffic that just takes without giving me anything in return.
how about this analogy: I created a most tasty cookie recipe. I give it out for free, and all copies have my name because I am vain person who likes to be known far and wide as the best baking chef ever. Is it ok to get the recipe, remove my name, and write in LLM-Codex as the creator? again, i'm ok with giving the recipe for free, i just want my name out there.
>Is it ok to get the recipe, remove my name, and write in LLM-Codex as the creator? again, i'm ok with giving the recipe for free, i just want my name out there.

From a legal perspective, it's a pretty clear "no". The instructions in recipes aren't copyrightable. The moral question is more ambiguous, but it's still pretty weak. Most recipes are uncredited, and it's unclear why someone can force everyone to attribute the recipe to them when all they realistically did was tweak the dish a bit. In the example above, I doubt you invented cookies.

i'm curious, do you honestly think the argument was about recipes and cookies? maybe it was an analogy? looking back up the comment tree, it does seem to be an analogy, not a discussion about ACTUAL cookies and ACTUAL recipes.
>maybe it was an analogy?

In that case it's a terrible analogy because if you can't get people to agree on the cookies case, what hope do you have to extend it to the case you're trying to apply the analogy to? It's like saying "You wouldn't pirate a movie, why would you pirate a blog post", because most people would pirate movies.

It’s interesting to see twists on the old anti-piracy arguments recycled for anti-ai.
Turns out many (most?) people on the internet were never anti-copyright in the first place. They were just anti-copyright (or at least, refused to challenge the anti-copyright people) because they wanted free movies and/or hated corporations.
Many of these people live int he countries where downloading for own use is lawful, since they're paying copyright levy exactly to cover for that.

They don't have to hate the copyright.

I will copy the supermarket and paste it somewhere else.

I'm also going to download a car.

That really depends, but the quick answer is that according to our human social contract, we'd just ask "how many can I take?". Until now, the only real tool to limit scrapers has been throttling, but I don't see any reason for there not to be a similar conversational social contract between machines.
Isn’t robots.txt such a “social contract between machines”? But AI scrapers couldn’t care less.
If someone hands out cookies in the supermarket, are you allowed to grab everything and leave?

Depends on the trust level of your society. where the store resides.

The internet is a cesspool of vagrants, thieves, mentally unstable, people and software with no impulse control, pirates and that is just talking about corporations. It gets so much worse with individuals.

This is a dishonest analogy. In your example, there is only a limited amount of cookies available. While there is no practical limit on the amount of time a certain digital media can be viewed.

You are allowed to take one cookie. But you are allowed to view a public website multiple times if you so want.

There sure is a limit in the load that the server you're DDoSing can take or the will for people to post new worthy content in public. The supply is limited just not at the first degree. Let's make a small edit: Are you allowed to take all the cookies and then sell them with a small ribbon with your name on it ?
Their is no arguing with pirates. They’ll take what’s yours and forget about you while you tend to the ashes.