[conradk]

You have to write 10 of them if they are not part of your workflow. If you TDD (which is applicable in a lot of cases in my experience with web development), then instead of testing with Curl or a web browser, you write tests. This takes the same amount of time upfront, so the additional cost of writing the tests is actually 0.

Not sure about other types of development, but in web development, TDD can be a good way to have automated tests without the additional cost.

[StavrosK]

Yes, if you've already written the tests, the tests will take no time to write. That doesn't mean they were free the first time around.

[conradk]

What I mean is that compared to testing in browser, writing unit tests and using those tests instead of the browser takes about the same amount of time. So doing tests in browser or in TDD ends up taking the same amount of time upfront. So testing in browser is as expensive as writing unit tests with TDD.

[collyw]

The whole tests don't take much time to write is bullshit. There are always things to configure in the test framework and something to go wrong. Plus you actually have to write the tests.

[conradk]

What kinds of things do you configure ? For us, the test environment is the same as the development environment, only we don't send out emails. Then there are some test scripts, but those are provided by Symfony and remain unchanged. So there is very little to setup.

And indeed, tests don't take much time to write once you get used to writing them. It's like anything. The more you do it, the better you get at it.

[dpark]

collyw is right. “Tests don’t take much time” is bullshit. Testing a feature typically takes as long as writing the feature, whether you pay that cost with unit tests, end-to-end tests, or ad hoc testing. (As hoc actually costs less up front but eventually many times more over the life of the product.)

If you’re very familiar with testing and/or doing TDD, you might include your testing costs in your estimate, but you still have the cost to pay. And if you write good testing up front, it will cost more up front.

[collyw]

The test environment on a local machine, as the tests are pretty useless if you can't run them. There can be database issues, as you aren't going to connect to the production database for running tests locally.

We have a full continuous integration environment at work, and the tests run there fine, but trying to reproduce that on a local machine is often a fairly difficult experience.

We have maybe 30 components in our system, so often I haven't touched the component before and I am asked to fix a bug in it. Sometimes they are using standard testing libraries, other times there are a lot of extra libraries that I haven't used before. Getting everything to play nicely isn't always trivial.

[dpark]

This implies your team needs to invest more in getting your local environment to parity. My team invested heavily in making the local setup identical to the pull request environment so that the testing is identical. We’ve also invested heavily to ensure that nearly[1] the exact same tests run in our continuous integration pipeline as our pull requests, with the net effect being that a break in CI is by definition an environmental difference.

This drastically scopes down the surface area for CI breaks.

[1] There are a few exceptions for tests that specifically cover environmental config/behavior that cannot be fully tested locally.

[szatkus]

If you're talking about emulating browser requests by tests you are already at integration tests territory.

[conradk]

Oh, I guess you're right. I may have interchangeably used unit-tests and integration tests because in PHP/Symfony, they look pretty much the same: some setup code and "assert()" calls.

Sorry for any confusion for anyone reading my previous comments.

[ljw1001]

You can isolate your server code behind an interface that lets you test it in your IDE, using the same requests that would normally originate in the browser. You just translate each browser request into an API call, and test that API call. No browser needed.

Cake + Eat-it, too.

[daanlo]

From my experience trusting a unit test instead of looking at things in the browser is a certain recipe for introducing bugs. Not sure if this is what you are advocating or if I misunderstood.

[conradk]

That is what I do for most back-end tasks (not so much for front-end stuff yet although there is some interesting work happening in the open source world for that too). And I've actually had far fewer bugs testing with TDD than testing in browser.

In the browser, testing edge cases sometimes requires custom headers, encoding of data to create authorization tokens, etc. This means you have to either have amazing browser tools (which I've yet to find) or use a combination of browser and shell to achieve what I need.

In TDD, most setup can be automated with simple function calls. In addition, well made frameworks, such as PHP's Symfony, have utilities that even avoid making real HTTP requests, so tests run faster than using a browser, but the result is the same.

I'm not saying everyone should do TDD, but from my experience, it can lead to increased productivity and fewer bugs in some cases.

By the way, if you know of browser tools that make testing easier, I'd be glad to learn more. I use TDD because I lack the browser tools. If I had the right tools, maybe I'd consider going back to in browser testing.

[gremlinsinc]

Have you tried dusk? It's for laravel, but it could possibly be used on other projects, depends on your stack I guess.

[conradk]

Didn't know dusk. Thanks for sharing ! It does look interesting ! And Symfony seems to be doing the same thing but for Symfony, called Panther.

[bluGill]

You still have to write the integration test to prove your units all work together.

[conradk]

I have been made aware further down that I might have confused unit and integration tests a bit. Sorry about that. In Symfony code, the two co-exist and look somewhat similar. Maybe that's why I don't really differentiate them.

[bluGill]

To be honest I don't actually differentiate them either.

When my function under test calls sort() I don't fake that call out, so technically I just wrote an integration test. (In fact I work with one group that will inject a fake sort())

If I write a library foo which has sub-module bar which has class baz and fuzz, is the test for bar that tests both baz and fuzz a unit test of an integration test? Of course if you are a user of my library tests for foo are unit tests to you...

[szatkus]

I once spent an evening reading and watching wise people on the internet to find out where is difference between unit tests and integration tests. It's really fuzzy and depends on a source.

There's no agreement what is "unit". Few classes interacting with each other could still be a unit.

I didn't find any strict definition which would be useful in practice. I just write tests and guess they are mostly integration tests, some end-to-end tests and a few unit tests.