I think it's because publishers have loosened up a bit on their requirements and self-publishing is easier than ever. Traditionally, publishers wouldn't really invest in putting out short books aside from some notable exceptions.
There's also the "too big to edit" feature of many bestselling authors - look at the size of the Harry Potter books as an obvious example: (https://www.barnesandnoble.com/w/harry-potter-paperback-boxe... third image). When an author gets famous enough they apparently get edited less.
There may also be something about people not wanting to pay "full price" for a "thin book" but I doubt that's a major factor.
Though I doubt many of the books on the NYT bestsellers list are self-published. In general though I agree with you. For a lot of non-fiction books 100 pages is probably plenty but, if you go through a publisher, you'll have to go longer. It's one reason I likely won't use a publisher again.
The book I put out through a publisher (Apress) was essentially open source software (How Open Source Ate Software) for the business person who understands it's important but really doesn't know where to get started to learn about it and how it works from a community, development, and business perspective.
I did feel I was stretching to fill some pages in the first edition. That said, I was asked to do a second edition and it actually ended up a bit longer. I cut some "filler" that I didn't think was essential, reorganized, and dove more deeply into some topics (like legal matters with the assistance of a lawyer colleague) and I think it is now reasonably in right length territory; it covers a lot of territory.
Another book I self-published on software packaging broadly and its analogs to various things in the real world today and historically ended up around 100 pages.