[mgkimsal]

> I have definitely been guilty of failing to test how my schemata behave with large data sets.

As have I. It's the dangerous part of the "get it done" approach. And there's no perfect approach - everything is a tradeoff. How much time do you spend dealing with situations that might never happen?

Experience does give you some grounding when making those tradeoff decisions. No, we don't need the architect the application to scale up/down to handle 25k concurrent users in 5 minutes; that's unlikely going to happen. Yes, we do need to spend the extra 2 hours installing and learning a templating system to avoid common XSS pitfalls.