Harry Potter should not be "free", but as a common cultural artifact it should be available according to the contract between the society and the author. For example, author is entitled to a share of profits from any distribution or derivative work, but the author is NOT allowed to decide who to allow and who to deny to distribute and derive it.
These days it is clear that copyright should only be granted by society in exchange to obligations to make the work available to the society at reasonable terms; as time passes the terms should be more and more permissive.
The copyright owners should not dictate their will to the society which empowered them in the first place when it's clearly hurts the society.
For example, I believe that every song recorded before 1997 should be available for purchase in any music web store (some regulated part of the revenue should still go to the musicians if they can be located, which is what recording industry association can do)
And all of those songs should be free to derive and cover (some small regulated part of the revenue on derived song should go to musicians via the same mechanism).
Or else old people would tend to become unavailable at all because you can't locate original authors. And music becoming available is very bad. Much worse than "scary criminal pirate" bad.
But it is related to a fact that copyright holders prevent people from convenient access to content in order to squeeze more short-term profits.
For example, tomorrow I'm going to pirate Game Of Thrones S02E01 because there is no way to obtain it where I live: legally, in English, with subtitles, tomorrow. Mind you, many people would still pirate it anyway, but I'm ready to pay, let's say, 5$ if it was possible. And I might even reconsider my position on piracy.
It's not so I'm going to pirate it and I feel no guilt because they've violated the contract: they provide content, I pay money.
The only way to fix the situation is legally force them to deliver. This way they lose some short term money, but they gain loyalty, crush piracy and win in the long term.
Same with music. Streaming services already eat at piracy; but not every musician is available on those. Same with books.
These days it is clear that copyright should only be granted by society in exchange to obligations to make the work available to the society at reasonable terms; as time passes the terms should be more and more permissive.
The copyright owners should not dictate their will to the society which empowered them in the first place when it's clearly hurts the society.