[workethics]

> That said ... putting part of your soul into machine format so you can put it on on the big shared machine using your personal machine and expecting that only other really truly quintessentially proper personal machines receive it and those soulless other machines don't ... is strange.

That's a mischaracterization of most people want. When I put out a bowl of candy for Halloween I'm fine with EVERYONE taking some candy. But these companies are the equivalent of the asshole that dumps the whole bowl into their bag.

[horsawlarway]

I really don't think this holds.

It's vanishingly rare to end up in a spot where your site is getting enough LLM driven traffic for you to really notice (and I'm not talking out my ass - I host several sites from personal hardware running in my basement).

Bots are a thing. Bots have been a thing and will continue to be a thing.

They mostly aren't worth worrying about, and at least for now you can throw PoW in front of your site if you are suddenly getting enough traffic from them to care.

In the mean time...

Your bowl of candy is still there. Still full of your candy for real people to read.

That's the fun of digital goods... They aren't "exhaustible" like your candy bowl. No LLM is dumping your whole bowl (they can't). At most - they're just making the line to access it longer.

[shiomiru]

> They mostly aren't worth worrying about

Well, a common pattern I've lately been seeing is:

* Website goes down/barely accessible

* Webmaster posts "sorry we're down, LLM scrapers are DoSing us"

* Website accessible again, but now you need JS-enabled whatever the god of the underworld is testing this week with to access it. (Alternatively, the operator decides it's not worth the trouble and the website shuts down.)

So I don't think your experience about LLM scrapers "not mattering" generalizes well.

[horsawlarway]

Nah - it generalizes fine.

They're doing exactly what I said - adding PoW (anubis - as you point out - being one solution) to gate access.

That's hardly different than things like Captchas which were a big thing even before LLMs, and also required javascript. Frankly - I'd much rather have people put Anubis in front of the site than cloudflare, as an aside.

If the site really was static before, and no JS was needed - LLM scraping taking it down means it was incredibly misconfigured (an rpi can do thousands of reqs/s for static content, and caching is your friend).

---

Another great solution? Just ask users to login (no js needed). I'll stand pretty firmly behind "If you aren't willing to make an account - you don't actually care about the site".

My take is that search engines and sites generating revenue through ads are the most impacted. I just don't have all that much sympathy for either.

Functionally - I think trying to draw a distinction between accessing a site directly and using a tool like an LLM to access a site is a mistake. Like - this was literally the mission statement of the semantic web: "unleash the computer on your behalf to interact with other computers". It just turns out we got there by letting computers deal with unstructured data, instead of making all the data structured.

[krupan]

"this was literally the mission statement of the semantic web" which most everyone either ignored or outright rejected, but thanks for forcing it on us anyway?

[horsawlarway]

I guess if my options for getting a ramen recipe are

- Search for it and randomly click on SEO spam articles all over the place, riddled with ads, scrolling 10,000 lines down to see a generally pretty uninspired recipe

or

- Use an LLM and get a pretty uninspired recipe

I don't really see much difference.

And we were already well past the days where I got anything other than the first option using the web.

There was a brief window were intentionally searching specific sites like reddit/hn worked, but even that's been gone for a couple years now.

The best recipe is going to be the one you get from your friends/family/neighbors anyways.

And at least on the LLM side - I can run it locally and peg it to a version without ads.

[w00ds]

It's crazy how appealing the irl version you mentioned is, compared to the online version. Looking through a book, meeting people and sharing recipes, etc. The world you're interacting with actually cares about you. Feels like the net can't ever have that now.

[shiomiru]

> If the site really was static before, and no JS was needed

One does not imply the other. This forum is one example. (Or rather, hn.js is entirely optional.)

> Another great solution? Just ask users to login (no js needed). I'll stand pretty firmly behind "If you aren't willing to make an account - you don't actually care about the site".

Accounts don't make sense for all websites. Self-hosted git repositories are one common case where I now have to wait seconds for my phone to burn through enough sha256 to see a readme - but surely you don't want to gate that behind a login either...

> My take is that search engines and sites generating revenue through ads are the most impacted. I just don't have all that much sympathy for either.

...and hobbyist services. If we're sticking with Anubis as an example, consider the author's motivation for developing it:

> A majority of the AI scrapers are not well-behaved, and they will ignore your robots.txt, ignore your User-Agent blocks, and ignore your X-Robots-Tag headers. They will scrape your site until it falls over, and then they will scrape it some more. They will click every link on every link on every link viewing the same pages over and over and over and over. Some of them will even click on the same link multiple times in the same second. It's madness and unsustainable.

https://xeiaso.net/blog/2025/anubis/

> Functionally - I think trying to draw a distinction between accessing a site directly and using a tool like an LLM to access a site is a mistake.

This isn't "a tool" though, it's cloud hosted scrapers of vc-funded startups taking down small websites in their quest to develop their "tool".

It is possible to develop a scraper that doesn't do this, but these companies consciously chose to ignore the pre-existing standards for that. Which is why I think the candy analogy fits perfectly, in fact.

[account42]

> They're doing exactly what I said - adding PoW (anubis - as you point out - being one solution) to gate access.

Which is a shit solution where everyone suffers.

> Another great solution? Just ask users to login (no js needed). I'll stand pretty firmly behind "If you aren't willing to make an account - you don't actually care about the site".

No I won't create an account to check if a search result has what I'm looking for. Not will I sign up to a forum before I know what the culture is like. We already had this shit with communities moving to Discord, we don't need fuck up the remaining web as well.

[igloopan]

I think you're missing the context that is the article. The candy in this case is the people who may or may not go to read your e.g. ramen recipe. The real problem, as I see it, is that over time, as LLMs absorb the information covered by that recipe, fewer people will actually look at the search results since the AI summary tells them how to make a good-enough bowl of ramen. The amount of ramen enjoyers is zero-sum. Your recipe will, of course, stay up and accessible to real people but LLMs take away impressions that could have been yours. In regards to this metaphor, they take your candy and put it in their own bowl.

[horsawlarway]

So what is the goal behind gathering those impressions?

Why do you take this as a problem?

And I'm not being glib here - those are genuine questions. If the goal is to share a good ramen recipe... are you not still achieving that?

[SamBam]

The internet would not exist if it consisted of people just putting stuff out there, happy that it's released into the wilds of the overall consciousness, and nothing more. People are willing to put the time and effort into posting stuff for other reasons. Building community, gaining recognition, making money. Even on a website like HN we post under consistent usernames with the vague sense that these words are ours. If posts had no usernames, no one would comment on this site.

It's completely disingenuous to say that everyone who creates content -- blog authors, recipe creators, book writers, artists, etc -- should just be happy feeding the global consciousness because then everyone will get a tiny diluted iota of their unattributed wisdom.

[horsawlarway]

How old are you?

I'm old enough I remember a vivid internet of exactly that.

Back when you couldn't make money from ads, and there was no online commerce.

Frankly - I think the world might be a much better place if we moved back in that direction a bit.

If you're only doing it for money or credit, maybe do something else instead?

> If posts had no usernames, no one would comment on this site.

I'd still comment. I don't actually give much of a shit about the username attached. I'm here to have a casual conversation and think about things. Not for some bullshit internet street cred.

[SamBam]

I'm more than old enough to remember the birth of the internet.

Back when I had a GeoCities website about aliens (seriously) it was still mine. I had a comments section and I hoped people would comment on it (no one did). I had a counter. I commented on other people's sites in the Area 51 subsection I was listed under.

The aim wasn't just to put out my same-ol' unoriginal thoughts into the distributed global consciousness, it was to actually talk to other people. The fact that I wrote it under a dumb handle (a variant of the one I still use everywhere) didn't make me feel less like it was my own individual communication.

It's the same for everything else, even the stuff that was completely unattributed. If you put a hilarious animation on YTMND, you know that other people will be referencing that specific one, and linking to it, and saying "did you see that funny thing on YTMND?" It wouldn't have been enough for the audience to just get some diluted, average version of that animation spread out into some global meme-generating AI.

So no, "Google Zero" where no one sees the original content and is just "happy that their thoughts are getting out there, somehow" is not something that anyone should wish for.

[reactordev]

You can’t bring back Compuserve.

You both are right however it’s the medium that determines one’s point of view on the matter. If I just want to spread my knowledge to the world - I would post on social media. If I want to curate a special viewership and own my own corner of the web - I would post on a blog. If I wanted to set a flag, setup a shop, and say I’m open for business - I would write an app.

The internet is all of these things. We just keep being fed the latter.

[jasonvorhe]

That's also trained behavior due to SEO infested recipe sites filled with advertorials, referral links to expensive kitchen equipment, long form texts about the recipe with the recipe hidden somewhere below that.

Same goes for other stuff that can be easily propped up with lengthy text stuffed with just the right terms to spam search indexes with.

LLMs are just readability on speed, with the downsides of drugs.

[lelanthran]

> I really don't think this holds.

Only if you consider DoS as the only downside.

As with this analogy:

1. I put out a bowl of (infinite and cost-free) candy, with my name written on each piece so people know where they got the candy.

2. Some other resident, who doesn't have an infinite and cost-free source of candy like I do, comes along and grabs all the candy at periodic intervals.

3. They then scrub my name from all the candy wrappers and replace it with their name.

4. They put out all the candy, pretending it is their candy.

This analogy is much more accurate than either mischaracterisation in this thread:

1. I have no objection to the other resident using me as an unlimited source of candy.

2. I object only to them obfuscating their source of candy, instead misrepresenting the candy as their own!

Because, you see, no one cared when search engines directed candy-hunters to your door. No once cared when search engines presented the candy with your name still on it.

The whole issue, which is unaddressed by your post, is scrubbing the attribution, and then re-attributing the candy.

[lblume]

> these companies are the equivalent of the asshole that dumps the whole bowl into their bag

In most cases, they aren't? You can still access a website that is being crawled for the purpose of training LLMs. Sure, DOS exists, but seems to not be as much of a problem as to cause widespread outage of websites.

[rangerelf]

A better analogy is that LLM crawlers are candy store workers going through the houses grabbing free candy and then selling it in their own shop.

Scalpers. Knowledge scalpers.

[horsawlarway]

Except nothing is actually taken.

It's copied.

If your goal in publishing the site is to drive eyeballs to it for ad revenue... then you probably care.

If your goal in publishing the site is just to let people know a thing you found or learned... that goal is still getting accomplished.

For me... I'm not in it for the fame or money, I'm fine with it.

[allturtles]

I think you're missing a middle ground, of people who want to let people know a thing they found or learned, and want to get credit for it.

Among other things, this motivation has been the basis for pretty much the entire scientific enterprise since it started:

> But that which will excite the greatest astonishment by far, and which indeed especially moved me to call the attention of all astronomers and philosophers, is this, namely, that I have discovered four planets, neither known nor observed by any one of the astronomers before my time, which have their orbits round a certain bright star, one of those previously known, like Venus and Mercury round the Sun, and are sometimes in front of it, sometimes behind it, though they never depart from it beyond certain limits. [0]

[0]: https://www.gutenberg.org/cache/epub/46036/pg46036-images.ht...

[b112]

It's a very simple metric. They had nothing of value, no product, no marketable thing.

Then they scanned your site. They had to, along with others. And in scanning your site, they scanned the results of your work, effort, and cost.

Now they have a product.

I need to be clear here, if that site has no value, why do they want it?

Understand, these aren't private citizens. A private citizen might print out a recipe, who cares? They might even share that with friends. OK.

But if they take it, then package it, then make money? That is different.

In my country, copyright doesn't really punish a person. No one gets hit for copying movies even. It does punish someone, for example, copying and then reselling that work though.

This sort of thing should depend on who's doing it. Their motive.

When search engines were operating an index, nothing was lost. In fact, it was a mutually symbiotic relationship.

I guess what we should really ask, is why on Earth should anyone produce anything, if the end result is not one sees it?

And instead, they just read a summary from an AI?

No more website, no new data, means no new AI knowledge too.

[horsawlarway]

I guess I don't derive my personal value from the esteem of others.

And I don't mean that as an insult, because I get that different people do things for different reasons, and we all get our dopamine hits in different ways.

I just think that if the only reason you choose to do something is because you think it's going to get attention on the internet... Then you probably shouldn't be doing that thing in the first place.

I produce things because I enjoy producing them. I share them with my friends and family (both in person and online). That's plenty. Historically... that's the norm.

> I guess what we should really ask, is why on Earth should anyone produce anything, if the end result is not one sees it?

This is a really rather disturbing view of the world. Do things for you. I make things because I see it. My family sees it. My friends see it.

I grow roses for me and my neighbors - not for some random internet credit.

I plant trees so my kids can sit under them - not for some random internet credit.

[b112]

Context. Note that we're having a discussion about people putting up websites, and being upset about AI snarfing that content.

> I guess what we should really ask, is why on Earth should anyone produce anything, if the end result is not one sees it?

>

> And instead, they just read a summary from an AI?

The above is referring to that context. To people wanting others to see things, and that after all is what this whole website's, this person's concerns are about.

So now that this is reiterated, in the context of someone wanting to show things to the world, why would they produce -- if their goal is lost?

This doesn't mean they don't do things privately for their friends and family. This isn't a binary, 0/1 solution. Just because you have a website for "all those other people" to see, doesn't mean you don't share things between your friends and family.

So what you seem to dislike, is that anyone does it at all. Because again, people writing for eyeballs at large, doesn't mean they aren't separately for their friends or family.

It seems to me that you're also creating a schism between "family / friends" and "all those other people". Naturally you care for those close to you, but "those other people" are people too.

And some people just see people as... people. People to share things with.

Yet you seem to be making that a nasty, dirty thing.

[Anamon]

You conflate doing something with sharing it online. A lot of people do things for themselves, then they post about it and share it because they like the idea of someone else enjoying and getting something out of it. The thing LLMs might get them to stop doing is not the doing of the thing, but the sharing, to the detriment of everyone who actually would have liked to see it.

And no, people sticking to the LLM summary won't get the ideas I shared. They get a crappy, broken, incoherent, messed-up, bland, averaged version of it. Purified of all the personality, insight and thought it might have had in it. That's why people getting an LLM summary partially derived from their data will never seem like a suitable replacement to someone who does it not for the views or credits, but because they actually want to share something of themselves.

I do agree that the solution would best come from the demand site. People realising the inherent blandness and horseshitness of LLM replies, especially when compared to something written by an actual human with thought and intent, ditch the low-quality LLM turds and demand real content again. The problem I see right now is that pretty much everyone would prefer the human version to the slop, but the megacorps force-feed the slop and spend billions trying to make it as inconvenient as possible to interact with other humans.

[shkkmo]

> But if they take it, then package it, then make money? That is different

But still, also legal.

You can't copyright a recipe itself, just the fluff around it. It is totally legal for somone to visit a bunch of recipe blogs, copy the recipes, rewrite the descriptions and detailed instructions and then publish that in a book.

The is essentially the same as what LLMs do. So prohibiting this would be a dramatic expansion of the power of copyright.

Personally, I don't use LLMs. I hope there will always be people like me that want to see the original source and verify any knowledge.

I'm actually hopeful that LLM reduction in search traffic will impact the profitability of SEO clickbait referral link garbage sites that now dominate results on many searches. We'll be left with enthusiasts producing content for the joy of nerding out again. Those sites will still have a following of actually interested people and the rest can consume the soulless summaries from the eventually ad infested LLMs.

[b112]

It may be legal in your jurisdiction, but I think this is a more generic conversation that the specific work class being copied. And further, my point is also that other parts of copyright law, at least where I live, view "for profit copying" and "some dude wanting to print out a webpage" entirely different.

I feel it makes sense.

Amusingly, I feel that an ironic twist would be a judgement that all currently trained LLMs, would be unusable for commercial use.

[lelanthran]

> If your goal in publishing the site is just to let people know a thing you found or learned... that goal is still getting accomplished.

I like how you posted so many times in this thread, with the assertion that that is the goal of people giving away stuff for free.

Your responses in this thread are almost textbook example of Strawman Argument; you could not do a better Strawman Argument even if you tried!

[CJefferson]

It's absolutely fine for you to be fine with it. What is nonsense is how copyright laws have been so strict, and suddenly AI companies can just ignore everyone's wishes.

[horsawlarway]

Hey - no argument here.

I don't think the concept of copyright itself is fundamentally immoral... but it's pretty clearly a moral hazard, and the current implementation is both terrible at supporting independent artists, and a beat stick for already wealthy corporations and publishers to use to continue shitting on independent creators.

So sure - I agree that watching the complete disregard for copyright is galling in its hypocrisy, but the problem is modern copyright, IMO.

...and maybe also capitalism in general and wealth inequality at large - but that's a broader, complicated, discussion.

[pas]

What people want is critically important, yet they are heart wrenchingly unaware how (web1.0) technology cannot give it to them.

And on a fundamental level - ie. as an information security problem - preventing copying by bots is logically incoherent with being open and free in the usual way.

The way to make progress on this problem is to realize this, and then let people either find alternatives to publishing on websites, start a movement to stop machine learning, campaign for accountable AI, work on getting political power and getting compensation from these companies, etc.

[reactordev]

More like when the project kids show up in the millionaire neighborhood because they know they’ll get full size candy bars.

It’s not that there’s none for the others. It’s that there was this unspoken agreement, reinforced by the last 20 years, that website content is protected speech, protected intellectual property, and is copyrightable to its owner/author. Now, that trust and good faith is broken.

[account42]

A yes of course, the poor poor AI companies getting scraps from the greedy independent website operators.