Hacker News new | ask | show | jobs
by lxgr 51 days ago
> it's now common knowledge that robots.txt isn't a hard requirement and can be avoided entirely, for example

Was it ever not? It's a text file, not law.

> They have absolutely turned the open web into a dark forest.

Only if you have an ideological problem with people you don't like using the things you publish on the open web.

I'd say the web can be very open even without being copyleft. It makes some business models non-viable, but it doesn't prevent anyone from publishing what they want.

On the other hand, I don't think I would call something that preserves copyright at the cost of only admitting "approved/certified non-LLM scrapers" via attestation or similar "the open web".

> Having a browser session able to be verified as untampered and/or "trusted" is probably going to be a thing going forward. Sucks a ton, but we all did this to ourselves.

Who did what to whom?