[nostril]

As far as I can tell, Books3 seems to be training data for language models. I'm not sure how it was created, but it contains a lot of books. I got it by torrenting The Pile after it was forced offline by the Danes.

They're asking to remove 109 books from the dataset, which I can do. But I'm not sure whether to. Once you set aside the question of law, it becomes a matter of ethics, and these questions aren't so easy.

[rmbyrro]

I wouldn't set the law aside. Do philosophy over the ethics if you want, but only after you are sure to have the legal side covered, because this one can ruin your finances and your life in general.

Unless you're based and incorporated in Iran, Iraq or North Korea, your country has signed the Berne Convention and has implemented in law some level of copyright protection that almost certainly makes the distribution of those books illegal.

If you're not taking very careful technical and legal measures to remain anonymous, you can get in serious legal trouble for breaking the law.

What is the upside for you? Companies like Uber, Google, etc break the law all the time. But they profit billions from that and then pay millions in fines and lawyers. What's your game? Are you profiting enough to make sense - financially-wise - to break the law?

Last but not least, I wouldn't play with lawyers' personalities trying to make them "please" you. Respect them, otherwise, they'll do whatever they can to make you regret it. And believe me, they can do a lot against you. These people are evil. Don't cross their paths.

[nostril]

> What's your game? Are you profiting enough to make sense - financially-wise - to break the law?

Not at all. Hosting costs $130/mo, and I feel the sting each month. I'm not sure we'll even get enough donations to cover that, let alone have some kind of profit motive. But we wouldn't want to profit off the works anyway, or else we'd be no better than the corporations.

My game is to help people like you be able to train your own models. If I don't help you, who will? Companies will have the final say in what you're allowed to do on your own hardware, because they control the data. No data, no training.

The hard part is to balance this with doing the right thing. I'd like to figure out the right thing from first principles and by asking thoughtful people like you, rather than from fear of consequences.

As for consequences, we're being careful enough that it seems worth the risk. (You can read more about our precautions at https://news.ycombinator.com/item?id=37346620.) But I agree that staying out of jail is preferable to being in one.