I recommend making a playlist and making a self-blind study which feels more random: your native music player shuffling it or a true random shuffling of the list.
That's because most people don't really want a "random" playlist. They want a random shuffle: A set of songs played in random order with no repeats until all have been played. That's not the same as making independent random picks from a list, but that's what most people want when they "randomize" their playlist.
I was more thinking about taking a list and shuffling it instead of picking songs at random.
In both cases, people complain because, for example, the same artist might be picked twice in a row. Most shuffle functions aren't random but try to make distance between similar songs.