[NathanKP]

You should consider testing to be a pyramid:

1) A smallish number of full end to end integration tests which actually make TCP requests to your app, and which actually have service talk to each other, etc.

2) A medium number of black box integration tests that just make HTTP requests to a single isolated service or component and verify that the response comes back out as expected (with no expectations on what happens in between input and output).

3) A much larger number of unit tests that are direct execution of business logic (just calling the code directly instead of calling the server externally) to verify various edge conditions in the inner underlying code.

To use a real world example you might have full end to end integration tests to ensure that your service allows the creation of a new account, and that when the account is created an email is dispatched, and that an authentication cookie is set which gives this new user the ability to make requests to the service.

This is just a high level test though. To back this test up you should have at least a couple hundred unit tests verifying that various edge cases with email formats, name formats, unicode characters in input, etc are handled correctly.

To speed up test suite execution time it doesn't make sense to run each of these hundreds of edge cases as full end to end integration tests, or you end up with a horrific test suite that will take minutes to run. So instead you should use unit tests to cover the underlying edge conditions. You can usually run a hundred unit tests in the amount of time that a single integration test will run.

[twic]

> You should consider testing to be a pyramid:

Why?

I've heard this story a million times. Nobody's ever been able to robustly justify it to me. I don't believe it's true.

[olefoo]

If you don't have a test for a given item of functionality; how do you know it works?

If you don't have a test for your subsystem that covers the more common use cases; how will you tell if it broke when you updated one of the functions/methods it uses?

If you don't have a test for individual flows at the level of the entire system how will you know when you did something that broke the build outright?

If you do none of that testing; you're handing off that function of your development team to the users of the system you're building.

Testing is an engineering practice that lets you make changes to complex systems with confidence that the system is still doing it's job after the changes you make.

It's not magic; it's engineering. As such, it's aim is to be boringly predictable.

And yeah, it's annoying; but it's a lot less annoying than facing an angry customer who is screaming about lost data, lost money and how they're going to sue you into a smoking hole in the ground for deliberately deceiving them about the issues with your software.

[iofj]

You're arguing in favor of unit tests.

> If you don't have a test for a given item of functionality; how do you know it works?

This would be an integration test

> If you don't have a test for your subsystem that covers the more common use cases; how will you tell if it broke when you updated one of the functions/methods it uses?

That would be a system test

> If you don't have a test for individual flows at the level of the entire system how will you know when you did something that broke the build outright?

I may misunderstand the question, but ... you compile it ?

> Testing is an engineering practice that lets you make changes to complex systems with confidence that the system is still doing it's job after the changes you make.

And if you have unit tests ... doing those changes is twice as hard as when you don't have them. Also if you test the "job" of the complex system, that would definitely not be a unit test.

This is what unit tests look like in large companies. What you actually find in the field

  def add1(a):
    return a + 1

  def TestAdd1(self):
    self.assertEquals(add1(2), 3)
    self.assertEquals(add1(-2), -1)
    self.assertEquals(add1(8), 9)
    self.assertEquals(add1(12), 13)
    self.assertEquals(add1(5557), 5558)

Think

None of your concerns apply to code like this. And nobody, ever, for any reason, should write code like this.

[olefoo]

While you can always find the degenerate case; it is somewhat of a strawman.

I personally don't get too hung up on whether a given test is a unit test, a system test or an integration test. For any reasonably sized project you're eventually going to have all of those. And sometimes something that was a unit test will suddenly become a test that crosses multiple units as requirements change and as dependencies get added to the project.

The viewpoint I take is that tests are an integral part of the software, they may not be part of the final delivered artifact but they are part of what defines the software.

You should be thinking as much about how to test that your code is doing what it should as to how to make it do so in the first place. If you do that you will have a far better understanding of the intent of any given piece of code and you will have done half the work of debugging before you write the bugs.

[twic]

Except maybe:

    self.assertEquals(add1(Integer.MAX_VALUE), ???)