[pietromenna]

You already got the main reason in your text: " engineers compromising in the face of business pressure or engineers making mistakes due to lack of experience or foresight"

Those two factors are a reality now a days: people leave teams when they get experience and go to a place where they have 0 experience (knowledge is lot) and they built everything under pressure to deliver to meet market demands (so we rush to deliver features).

I would also tell you that in the past well organized projects were exception not the rule.