Some cultures eschew dairy almost entirely. Most of the world is lactose "intolerant" actually. Not sure why it would be considered an important part of ones diet.
Dairy is a key part of the diet of hundreds of millions of vegetarians. It's undeniably important to the diet of a large portion of India's population, for example.
A different sort of vegetarian diet might work without any dairy (I'm pretty sure this describes some traditional Buddhist food). Not everyone uses the same stuff.
And Inuits have tradirionally had an essentially carnivorous diet. Diets have adapted to niches from long ago - even though lactose tolerance is technically the mutation instead of lactose intolerance.
A different sort of vegetarian diet might work without any dairy (I'm pretty sure this describes some traditional Buddhist food). Not everyone uses the same stuff.