[barrkel]

The things that I remember XP being controversial for were pair programming and TDD - real TDD, where you write the tests first and let the tests drive the design. And those are the two things that I don't really see as having caught on.

I mean, pairing is a fine approach when training someone up on a codebase, but it tends to be much more effortful for the guy in the driving seat, while the guy looking over your shoulder is making small comments and doing researchy lookups. This makes it less than efficient when both people are at the same knowledge level. The extra eyes looking for bugs is debatable; the bugs are more thoroughly found with tests.

Test-first TDD is even less popular. Norvig vs Jeffries was enlightening - http://devgrind.com/2007/04/25/how-to-not-solve-a-sudoku/ .

Software does have a much heavier focus on testing than it used to, to the point that in many projects, everything is implemented twice - all features have two representations, one in the form of the implementation, another in the form of tests, and often with the lines of test code outnumbering the implementation.

But other things have suffered IMHO; making code easy to test tends to over-abstract it, making it more parameterized and exposing more implementation details of high-level abstractions.

APIs are often uglier with a lot more exposed symbols to handle the parameterization, with various bits and bobs asking for interfaces that only have one concrete implementation that can only be created by a factory, and you have to learn the knack of actually instantiating the useful bits anew for each library. I've got Java squarely in mind, of course, and I'm convinced better language design can solve the problem with less harm to the software.

[henrik_w]

I don't agree with "making code easy to test tends to over-abstract it". The dependencies are there no matter what, but they become more obvious when you try to test the code in isolation. I think the main benefit of TDD is the way it forces you to break the program apart in smaller, less dependent pieces. The end result is better structure and less dependencies between parts.

[barrkel]

I fundamentally disagree with you.

If the goal in writing software is to reduce it to a set of pieces that plug together, I'd agree. But it's an arbitrary metric and not an absolute measure of quality, not by a long shot.

Note that I don't say "composable", because composability is something that needs to be designed in, and it isn't usually clear how to do it best until the third or so time around - the rule of threes.

Furthermore, I don't say reusable, because reusability is something that also needs to be designed for. In particular, reusability in such a way that software can evolve over time without breaking clients (reusers) of the abstraction demands a tight and constrained contract that is broadened carefully, while testable components demand broad and flexible contracts, otherwise not everything would be available to be tested.

Every part that testing has forced you to break out to be individually addressable is a part that you cannot remove in a refactoring that significantly changes the way a library solves its problem.

A library that has been broken into parts that are neither composable nor reusable is simply over-complex. Almost every extant Java library is like this!

Of course, if you just write end software in small teams, none of this is relevant to you. But it is crucial in library design, especially when client code is outside of your organization.

[Peaker]

If you split it into little parts, and those parts have no sensible meaning on their own or no sensible interface and semantics -- then you're absolutely right. Testing such parts will be difficult, too, because of this.

But if instead of a monolith, you have a set of components with well defined interfaces that have simple semantics (that do not leak abstractions) -- whether or not these parts are re-usable in other contexts -- then you almost automatically have higher quality software:

* Easier to test means it will likely be better tested

* Well-defined interface and abstraction and a small implementation means that reviewing/correctness becomes easy. You only need to understand a small component to review it. "Obviously no bugs" rather than "no obvious bugs"

* Easier to split the work across developers

* Easier to comprehend the whole as a collection of its parts

The total number of lines of code, or even the total complexity may increase relatively to a monolithic design. But correctness becomes so much easier.

You mention refactoring, too, and IME, refactoring can be both easier or more difficult, depending on whether it is within a single small component or across multiple ones.

If you add architectural/design changes -- then it is night and day. A monolith will likely have to be rewritten to make an architectural change reliably.

A set of components can easily be split, for example, so that a few components are moved to run on a different system with a network protocol between them.

[henrik_w]

A program made up of loosely coupled pieces has several advantages over one that is more monolithic. It is composable – all you have to do is put the parts together, then you have your complete program. It is easier to modify, because parts can be swapped out with minimal impact. It is easier to test, since the parts can be tested in isolation, which makes testing the complete program much easier.

As for reuse, I like this quote: "Don't aim for reuse. Write small, independent components you can reason about, and the right pieces for reuse will fall out." Jessica Kerr @jessitron on Twitter

[taeric]

How did that work out for the Kernel debates? :)

I don't think anyone disagrees that a "well designed and written" program of loosely coupled pieces has advantages over a monolithic one.[1] The debate is really over which is easier to do. And, the argument you are responding to is essentially, that it is easier to abstract out parts after you have done it in whole a few times. I know, personally, that that is a very compelling argument.

[1] Well, there probably is some debate on the feasibility of making things as loosely coupled as you would like. Back to the kernel debate, how many microkernels have survived with the device support of linux?

[spand]

The kernel debate is a different one.

Software can easily consist of loosely coupled pieces as source code and be compiled and run as a single monolith with hardly any performance loss at runtime (versus coupled source code).

[taeric]

Maybe I misrepresent the debate, then. My understanding is that the debate was that there was no future in a monolithically sourced and run kernel. Linus took the position that while that had a certain appeal, he just wanted an operating system he could use. If anybody had managed to deliver on the microkernel dream, he would probably not have started the linux kernel.

That is to say, that there is appeal to the "loosely coupled" dream of a software solution. Not just in source but in execution. However, there is the reality that this is very hard. The contention in this thread is that to think you can start in the loosely coupled set of parts is very ambitious.[1] It isn't that it is a bad goal. Just that it is akin to wanting to score well in a marathon without first running a few smaller races.

[1] Unless I am misrepresenting that, of course.

[ollysb]

One of the biggest problems I find with people adopting TDD is that they create one test per class. This was never the intention. The idea was that you test units, which may be a single class but equally may also be an aggregate made up of smaller classes. Tests should pin down external behaviour without overly restricting internals. Mocking at every class boundary leads to brittle tests suites that aren't focused on APIs, not to mention being a PITA to refactor.

[hderms]

I really appreciate your response. It's very measured and, to me, seems the result of experience, as opposed to gung-ho idealism. People need to realize that software is complicated, and that despite the fact that computers operate on unambiguous rules, that the people creating the software often can't depend on concepts in the same way. Following test-driven development fanatically is not a shortcut to designing re-usable, composable software. There is no shortcut to that.

[heleph]

Pair programming is supposed to be two people thinking through the problem together. It works when both people are immersed in solving the problem together.

When it works well, having two brains working together brings the benefit of different perspective and experience. It gives the opportunity to riff off each other's ideas. You spot issues with design and implementation earlier because having to communicate your ideas means working harder on them before you try to turn them into code.

I find it fun to work with someone else who is smart and engaged. It's magic when you become warmed up enough that things really start to flow. I've actually managed to get into flow before while working with someone else.

That said, it's pretty difficult to get right. I found it quite hard to let someone else see my process. If both sides aren't engaged in problem solving it can be really boring. I've also found that it takes me a while to figure out how to work productively with new people. The dynamic between any particular pair of people is a bit different. I think you need to build trust with your pair.

[henrik_w]

A "simpler" version is discussing the problem with your colleague in front of a whiteboard (drawing almost always seems to help). Once you've worked out how to implement it, you go off and implement it yourself.

[crdoconnor]

>But other things have suffered IMHO; making code easy to test tends to over-abstract it, making it more parameterized and exposing more implementation details of high-level abstractions.

I think this highlights a deficiency in testing tools. It's quite hard to, for example, change the system time when running a test which often means that you have to abstract out that part from the method you're testing and pass it through as a parameter.

You shouldn't need to do that (in python you don't!).

Also, there is a lot of APIs out there with very real world effects and integrations that it is pretty cumbersome to build mock ups of. Most API providers also just don't.

Mocking what would happen, say, when a twitter oauth token expires, isn't as easy as it should be.

On the plus side, UI testing tools seem a lot better nowadays.

But yeah, there's a serious dearth in good testing tools and bad language design (cough Java) that ends up making code unreadable.

[tmoertel]

> It's quite hard to, for example, change the system time when running a test which often means that you have to abstract out that part from the method you're testing and pass it through as a parameter.

Passing the time into your function isn't necessarily a thankless chore, however. It's actually quite similar to strengthening your induction hypothesis when doing a proof. Now your function doesn't just claim to work correctly for one time (the implicit clock time) but for all times. This stronger claim (if true) makes it easier to reason about the code that relies on the function, including not only the testing code but also the rest of your application code.

[henrik_w]

The question of time in relation to tests is interesting. If you write the code with the mindset that you should be able to test it (including time dependent behavior), you can end up with testable code without too much trouble. The key is to make time external to the code. I just blogged about this in "TDD, Unit Tests and the Passage of Time" http://henrikwarne.com/2013/12/08/tdd-unit-tests-and-the-pas...

[sgt]

Regarding pair programming, I also see it as paying the salaries for 2x developers, yet gaining very little from it. Productivity may be even less than from a single programmer, at least in my experience.

To elaborate, I've tried pair programming myself and it was completely inefficient when we tried it. I'm not going to dismiss it entirely though, perhaps we approached it the wrong way. Personally I just need a bit of space before I can start focusing in-depth about certain problems.

This is also why I like to be well-prepared before attending team design decisions, because coming up with good ideas "right there and then" is difficult for me.

[JackMorgan]

I think of pair programming like dancing. How much practice does it take to be able to dance with a partner before it's natural? More than a week, that's for sure.

I pair on all production code at work with only two other guys. I've worked with them for the last year. Together, any combination of the three of us is easily twice as effective as the fastest in our team. Something about the rhythm of the session, alternating roles, support when tackling boring parts, and the camaraderie frees us up to just get stuff done.

But, we work in a very complex domain that, a year in and many seminars by product later, we only barely are starting to grasp, with a large difficult to grasp system, sometimes solving problems just outside our comfort zone. It's not just CRUD and forms. So, maybe pairing is the four wheel drive of the programming world: uses more gas on the highway, but depending on your terrain, it might be the most fuel efficient way to get across a mountain.

[klibertp]

> I've tried pair programming myself and it was completely inefficient when we tried it

Or: I've tried Vim and my writing/editing speed halved. Or: I tried APL but it took me half a day to write one line of code. Or: I tried playing guitar and it sounded horrible.

I get the feeling that maybe the outcome would be different if you'd try doing it for a while longer. No guarantees, though.

[cs02rm0]

Regarding pair programming, I also see it as paying the salaries for 2x developers, yet gaining very little from it.

I think the trick is to use it when developers feel necessary to stay productive. No point having someone slogging away at something they find difficult and frustrating if a second pair of eyes and maybe some more specific knowledge of the area can help.

[beersigns]

I had similar experiences with pairing; often it was mandated by folks in charge who didn't seem to have a great grasp of what the benefits were. Productivity/velocity tended to suffer noticeably on many teams. It worked for others, but I think most times it wasn't understood fully why it worked for a given team.

What bothered me about how I saw pairing used was that people seemed to make blanket assumptions about it's benefits. Many times I saw people pairing up on trivially easy tasks. Seemed to me that pairing was a lot like everything else, it can be done well or poorly.

Pairing should be a naturally occurring process IMO; ie I don't know how to best accomplish a task or 'story', so I ask a team member w/expertise or experience to help point me in the right direction. If I need help beyond that, it becomes a pairing/knowledge-transfer exercise. I came to refer to it as "informal pairing". In general I tended to gravitate toward pairing on the exceptionally difficult tasks or ones that would have far reaching design implications.

[pjmlp]

> Test-first TDD is even less popular

I never bought into TDD, although I do write unit tests for most of the code where it makes sense.

No one has been able to show me an usable way to do TDD when coding native UIs, mobile OS, embedded systems or when using third party libraries not built with testing in mind.

Plus TDD makes very hard to properly design algorithms and data structures, that should beforehand be done at the whiteboard.

[jerf]

"No one has been able to show me an usable way to do TDD when coding native UIs, mobile OS, embedded systems or when using third party libraries not built with testing in mind."

I agree. However, I do not consider this a strike against testing; I consider it a strike against native UIs, mobile OSs, embedded systems, and third party libraries that don't support testing. You may not do TDD (I generally don't), you may not strive for 100% coverage, but testing is a fundamental aspect of serious software engineering, and anything that actively fights your attempts to test it is a big strike against that tech. I only use the ones that fight you that hard because there's unfortunately no competition, but it's still a disgrace. In 2013, testing ought to be a fundamental first-class concern of any new UI library, yet here we are.

[kabouseng]

Have a look at test driven development for embedded C by James Grenning for a viewpoint of how TDD might work in the embedded world. I found it to be a great resource.

http://pragprog.com/book/jgade/test-driven-development-for-e...

[dkersten]

This is a fantastic book!

[pjmlp]

Thanks, I know that book.

[joshuacc]

Why do you think that TDD prohibits you from using a whiteboard to design your algorithms and data structures beforehand? As far as I can tell, doing that work, then writing your first test is just TDD done well.

[pjmlp]

Because that is what TDD advocates sell at agile conferences, design by coding.

[dfox]

I practice what essentially amounts to pair programming very regularly. On the other hand any kind of TDD seems to be mostly moot for projects I work on. I assume this is because most of my projects involve disparate components with continuously changing interfaces and just getting together works better than producing interface specification that is complete enough to base any tests on (and to be clear: it's not about two people being in same room and each hacking away at his component, but about both working alternatively on each component).

I'm somehow surprised that this even works well with pairing of programmer and hardware engineer, but that requires management that believes in their engineer's skills (rapid iteration and hardware tends to get very expensive very fast) and programmers that have meaningful insight into hardware.

On the other hand projects I work on are probably not very representative of anything as most of them are weird :)

[huherto]

My recommendation on pair programming is that it is a good idea, you should try it, you should do it 5%, 10%, 20% of the time, whatever you want, I don't think is a good idea to do it 100%. I see two main benefits on pair programming.

  - Knowledge transfer. Learn new and better ways of working. IDE usage, short cuts, etc.

  - On a complex piece of software is better to have two sets of eyes checking everything.

TDD. I enjoy doing it, I am not strict on doing test first, most of the time I don't. I usually shoot for 70% coverage. Indeed, the tests take a significant part of the effort, often they are brittle and you need to refactor them, but I really think they improve your overall design, your confidence on the robustness of the system.

[mathattack]

I was thinking the same thing reading this. I don't see many instances of pair programming. Granted, my data points are limited, but that seems to be the exception, not the rule.

I've seen more people extoll (and consultants sell) automated testing than folks actually use it, let alone TDD. As an idea, I get it, but the implementation still seems mixed.