[dan_manges]

Dan from Braintree here. I think a QA team would be valuable; we just don't have one right now. To answer your rhetorical question, regression bugs aren't the only type of bug, but they're one of the most dangerous in a payments system. If there's a bug with an unanticipated use of a library, it will show up in our sandbox environment before merchants hit it in production. But if functionality that works in production breaks because of a change, that's a really serious problem.

[yellowredblack]

How are regression bugs more dangerous than other kinds of bug?

Here's how I'd rate the "dangerousness" of a bug:

  1. How easy is it to detect?
  2. How easy is it to reproduce?
  3. What are the consequences of it occuring?
  4. How likely is it to happen?

Look, bravo for all the TDD. TDD eliminates a huge chunk of bugs. But by definition, the bugs that you find with CI are easy to detect, easy to reproduce and 100% likely to happen. Sure, without a TDD/CI system, these bugs may not have been detected, may not have been easy to repro. But the reverse does not hold: a TDD/CI system doesn't make all bugs easy to detect and easy to repro.

So all the other bugs that your system has right now, are the ones that are left: hard to detect, hard to reproduce, and don't always happen. Now turn on a thousand users. How many users are you hoping to have btw?

Your worst kind of bug:

  * Is not detected for months.
  * Unable to reproduce.
  * Company killer. (Reputation, lawsuits, whatever).
  * Happens once every 40,000,000 sessions.

Not detectable using TDD and CI. Company still dead.

[hello_moto]

And QA would be able to detect these worst kind of bugs?

I'm not suggesting that QA is useless, I think QA should guide developers in terms of testing, as in QA should help writing the test-cases including the corner-cases in spec and let the developers write more tests around those things.

I also think that QA should help performing benchmark tests, load tests, and probably write end-to-end automation-tests (what do they call it these days? Acceptance tests?)

Last but not least, QA should redefine the software processes if bugs happened regularly in a particular area. Consider QA to be a manager that responsible for the productivity of your software team: if a software process doesn't work (let's say one day you found out that TDD doesn't work well), QA should detect that and figure out a better way.

Unfortunately, QA these days are still old school button clicker and test-case fanatics (i.e.: prepare 1000 test cases and ask the director for a week to run them all).

But at the end of the day, bugs exist. No amount of human or practices would cover those exotic bugs.