[mindslight]

What you're finding with A/B testing is local maximums. Which might make sense when your sole goal is squeezing a few more pennies out of a mundane business. But generalizing this to all of existence is sorely mistaken.

Asking "why" is the act of cognition. Discerning structure, compression, building a model (science). Then applying that model to new territory (ie engineering), to achieve better gains than undirected walk. Eschewing this is basically nihilism, and it's far too common these days.

[edmundsauto]

Asking why can lead down two paths. In a system where it's possible to understand the "why" (often overlaps with "hard" sciences), it is crucial.

In fields where the inputs are unknown, obscured, non-linear, and sometimes non-deterministic: asking "why" is going to create cognitive biases that are very difficult to break.

- Why did the Soviet Union fall?

- Why did Trump lose in 2020?

- Why did I eat a cheeseburger when I'm trying to lose weight?

Those all turn into narratives that may or may not be correct, but are usually self-reinforcing. Confirmation bias also means we evaluate the "why" question as follows: "given what I know about this situation, and how I think things generally work, can I see this singular narrative as being true?" That leads to blindness of other possible interpretations, as well as other cognitive dysfunction.

To be a little cheeky - that is why commentators on politics, economics, social sciences, psychology... are full of shit.

Asking why may be the act of cognition, as you say - but my take is that the act of cognition itself will not lead to understanding the truth, or better decisions. Making up fairy tails is also cognition but does not help us build mental models to better understand the world around us.