[yellowredblack]

Testing: testing is at the forefront of our development philosophy. We never need to check our code coverage to know that it's at 100%: with disciplined TDD, no line of code will be written without a test.

Bravo. In my experience that can be overkill, but with finance, I agree: why risk it. TDD everything.

We don't have a QA team.

WTF?

That might be terrifying when you consider the type of software that we're building, but we're confident that our automated testing is thorough and will catch any regression bugs.

Are regression bugs the only kind of bugs?

We use continuous integration to test every version of every client library against our gateway.

What happens when someone uses your client library in a way you didn't anticipate?

What happens when johnny-botnet hits your API directly without using the client library?

I spent several years developing games, with QA teams that outnumbered the developers. The QA team did not just play the levels through and say "it works!". Sure, they did that for the first hour. Then they'd start doing all those things that they thought someone might try (e.g. in a fit of boredom, or for a laugh). After that they'd just try breaking stuff. What a lot of bugs they would find!

As I write I find it hard to believe that I, a game developer, is having to explain the importance of QA to a financial company.

[aquark]

Agreed -- 100% code coverage is impressive, but only means you are testing the code you've written.

What about all the cases you forgot about. By definition you forgot about the tests too.

What about all the interactions between module A & module B? They may work 100% on their own, but not together.

A good QA department can generate many more tests that the developer could, and do so without the 'author blindness' that is somewhat unavoidable when writing code.

[dan_manges]

Dan from Braintree here. I think a QA team would be valuable; we just don't have one right now. To answer your rhetorical question, regression bugs aren't the only type of bug, but they're one of the most dangerous in a payments system. If there's a bug with an unanticipated use of a library, it will show up in our sandbox environment before merchants hit it in production. But if functionality that works in production breaks because of a change, that's a really serious problem.

[yellowredblack]

How are regression bugs more dangerous than other kinds of bug?

Here's how I'd rate the "dangerousness" of a bug:

  1. How easy is it to detect?
  2. How easy is it to reproduce?
  3. What are the consequences of it occuring?
  4. How likely is it to happen?

Look, bravo for all the TDD. TDD eliminates a huge chunk of bugs. But by definition, the bugs that you find with CI are easy to detect, easy to reproduce and 100% likely to happen. Sure, without a TDD/CI system, these bugs may not have been detected, may not have been easy to repro. But the reverse does not hold: a TDD/CI system doesn't make all bugs easy to detect and easy to repro.

So all the other bugs that your system has right now, are the ones that are left: hard to detect, hard to reproduce, and don't always happen. Now turn on a thousand users. How many users are you hoping to have btw?

Your worst kind of bug:

  * Is not detected for months.
  * Unable to reproduce.
  * Company killer. (Reputation, lawsuits, whatever).
  * Happens once every 40,000,000 sessions.

Not detectable using TDD and CI. Company still dead.

[hello_moto]

And QA would be able to detect these worst kind of bugs?

I'm not suggesting that QA is useless, I think QA should guide developers in terms of testing, as in QA should help writing the test-cases including the corner-cases in spec and let the developers write more tests around those things.

I also think that QA should help performing benchmark tests, load tests, and probably write end-to-end automation-tests (what do they call it these days? Acceptance tests?)

Last but not least, QA should redefine the software processes if bugs happened regularly in a particular area. Consider QA to be a manager that responsible for the productivity of your software team: if a software process doesn't work (let's say one day you found out that TDD doesn't work well), QA should detect that and figure out a better way.

Unfortunately, QA these days are still old school button clicker and test-case fanatics (i.e.: prepare 1000 test cases and ask the director for a week to run them all).

But at the end of the day, bugs exist. No amount of human or practices would cover those exotic bugs.