[grandalf]

The issue of where to draw the line between developer testing and QA testing is an interesting one.

I think it's best to answer it in reverse and start with a bug.

Did the bug make it to production? Is it a critical path? Was it covered by the test plan? Was it covered by a functional/unit/integration test? Was it clearly defined in the product requirements? Was the behavior clearly defined as part of a business objective?

Most bugs could have been discovered earlier in the chain. The earlier the better. Many times bugs are a result of systems failure, bad communication, assumptions, etc.

I think the two most important concepts in QA are 1) answering the question of "can we ship this now?" and 2) encouraging systems thinking so that all of the processes related to executing business goals in a software company can be continuously improved upon.

[el_benhameen]

I manage a QA team where there was none until very recently. If you have any reading on point #2 that you'd recommend, it'd be hugely helpful to me.

[grandalf]

I'm not aware of any specific reading... but I'll share a few more thoughts:

The key is to not make it a blame game. Every bug that is discovered on staging or production should (in theory) trigger a root cause analysis of some kind... because it means that one of the earlier processes failed.

If you have manual QA people, it's mostly just improving the QA plan and adding edge cases and domain knowledge, but it can also sometimes be that there are data bugs or integrations on production that are not on staging, slightly different on staging, etc.

With every bug there is an action plan for how to fix the bug, but that is separate from the knowledge that we gain about our system because the bug occurred. Maybe the fix is to get someone to manually clean up some dirty data that wound up on production due to someone forgetting to validate/clean input data ... that's great but from the QA perspective you might have learned that several of your steps failed.

So I think the main thing is considering bugs opportunities to learn about the system as a whole. The philosophy behind chaos monkey at NetFlix is that even a well tested, solid system needs to be resilient, so any opportunity to make your system stronger (regardless of the cause) is a good thing. In particular, any bug found before it hits production is a win overall.

I'd also add that it's important to let the knowledge flow back out of QA and into the product team, etc. QA people often end up becoming internal domain experts who catch lots of issues, but that is something that quickly exceeds what one person can remember/understand as a system scales, so organizational learning/practices pay off big.

[el_benhameen]

I really appreciate your comment. One of the more exciting parts of expanding has been getting the opportunity to think about systems and processes at a higher level now that I'm not the only one fighting fires. I'll take this advice to heart.

[bklaasen]

The two books on systems thinking that I'd recommend are Gerald Weinberg's "An introduction to general systems thinking" and Donella Meadows' "Thinking in systems: a primer". Anything by Gerald Weinberg is eye-opening, if you haven't come across him before. He's the grand-daddy of modern thinking on testing.

If you follow the advice in this article, you'll end up with an inefficient old-school, documentation-heavy, dogmatic factory-style quality-police testing department, set up in opposition to your development team.