[porker]

So a CMS that works as the client requires and has a very nice UI can't be taken seriously because it has no tests?

That's a very developer focused mindset.

I'm glad it's been open sourced and I hope to see tests added by them or the community, but it's not a deal-breaker for me.

And if you contribute tests I hope they'll be full functional tests to spot cross-browser regressions, JS breaks and the rest ;)

[jlafon]

Yeah, it's probably a developer focused mindset, but I'm a developer.

Without a test suite, it makes it very difficult for anyone to contribute, because there isn't a clear way to tell if they broke something. How do you decide to accept a pull request?

Then there is the entire issue of security. Without proper tests, how can you test that login/logout/sessions work? How can you tell if the csrf token is present? The list goes on.

What about databases? Does it work with Postgres? MySQL? How do you know?

I'll say it again, I think it's a cool project - but it needs tests. Tests are a developer's best friend.

[collyw]

Do you TDD kids never try to read and understand the code? Just assume because it passes a test it is working?

[batiste]

Clients do care about having a stable and bug free software. Tests is one of the best way to achieve this.

[porker]

I know and I agree, but I feel tests often cover the wrong thing, or what the developers feel is most important. Unit testing libraries within the system: great. Leaving the rest: not so great. Or aiming for 100% code coverage: uggh.

Playing devil's advocate here: Testing the system throughout to ensure components play nicely together and that functionally the system works as expected - this is in my opinion more important than unit testing.

[SEJeff]

There are two types of tests.

unit tests - most common and are tightly tied to the code. integration tests - simulate example user behavior and test the pieces work together as expected. For this type of app, selenium is the best tool for the job and splinter is an amazing python abstraction ontop of selenium. For instance, I have Jenkins setup splinter to run chrome and Firefox tests headlessly using a frame buffer (xvfb) to test the apps I'm responsible for as part of $dayjob.

Both types of tests are necessary and equally important.

http://splinter.cobrateam.info

[porker]

I applaud your view. Here in the PHP universe we're only starting to exit the "Everything must be unit tested" phase of programmer evolution, and integration/functional testing is still a novelty.

I've not come across Splinter before, that looks interesting. I've been using PhantomJS via CasperJS but my tests so far have been brittle...

[SEJeff]

Well we used to be in the same boat and then... A new developer wrote a few new bit pieces of code that were supposed to work together. He overly unit tested each piece discretely, but they didn't work together. So when we pushed it out to staging and I manually tried to look at things, the entire design was wrong and would never work.

From then on, the next task said individual worked on was setting up selenium and write integration tests. Now we are in a pretty sweet place.

We have unit tests, which verify each discrete thing works as it is expected to and make refactoring large swaths of code a breeze. Then we have integration tests, which run on IE (in a windows vm that is on our build server), firefox, and chrome via xvfb. It even helps us catch those obnoxious conditions where javascript runs fine in "real browsers" and fails on IE's lameness. Give it a try :)

[collyw]

According to this, it doesn't help that much compared to other things.

http://kev.inburke.com/kevin/the-best-ways-to-find-bugs-in-y...

[batiste]

Interesting article. Although I doubt most project will have 1000 beta testers ready to fill up detailed bug reports or some benevolent code inspectors.

A couple of simple integration tests go a long way detecting the most visible bugs.