"leisure" was probably the wrong word, in context. The sort of 'leisure' (or pastime) of watching TV can be completely passive. One can 'watch' TV for 2-3 hours and literally not have to think about anything difficult. Reading a book is far more of an 'activity - requiring active thought processing - than TV watching. And for many people, reading a book is anything but pleasure.
My main argument is that reading books is the hard part. Leisure time is an example I gave of a contributing factor. There are other contributing factors, such as cultural ones, which I alluded to.
Beyond that, though, it's not merely the case of an individual having time to read. A parent who does not read to their child is raising a child to be less literate or even illiterate.