The only difference between 1-2 star rating (using it as a metaphor) people and 3-4 rating people might just be a difference in opinion about how nice they should be or just how to normalize their score.
(Author here) Yes this is very true. The article is a bit of an oversimplification. There were 1-2 star people that we listened to, and 3-4 star people that shit on the book, and 5 star people with good constructive feedback.