[Veen]

Yes, it's mainly because Google started using a similar heuristic a few years ago: length equals quality. Or so the SEO world concluded. It may not have been true; correlation != causation is not a popular concept in SEO. The result is publishers and marketers padding content to achieve some imaginary ideal length. It's why recipes, for example, have hundreds of words of irrelevant crap at the beginning.

That said, the long-form essay is an ancient genre—much older than the modern magazine article—and they have always been filled with "irrelevant" asides, tangents, artistic flourishes, and so on. Concisely conveying their "central message" wasn't the primary point of the form, and people who enjoyed that type of essay wouldn't expect a linear explanation of the topic. The prose style, imaginative complexity, unexpected comparisons, digressions, and explorations were integral to the genre.

Sadly, great essays of that type take a long time to write and edit, and most people aren't interested in reading them. So we get long, repetitive, unimaginative junk instead.