[nostrademons]

I don't think it's incompetence, at least not at the academic/professor level we're talking about. Most people who get to be professors at research universities understand the low-level details of performance quite well.

Rather, I think it's often because the details about how a system will be used aren't clear until the system is built and people actually use it. As you know, engineering involves tradeoffs. It's possible to make one operation run faster by making another slower, or it's possible to make the system easier to use by making them both slower. Academics - and startups - aren't in a position to make those tradeoffs, because they don't know how the system will be used. So they'll micro-optimize along one dimension, but it turns out that people really use the software in a completely different dimension.