[nakor]

What if I want to test some part of the function in isolation? At my current job I have to maintain a huge and old ASP.NET project that is full of these "god-functions". They're written in the style that Carmack describes, and I have methods that span more than 1k lines of code. Instead of breaking the function down to many smaller functions, they instead chose this inline approach and actually now we are at the point where we have battle-tested logic scattered across all of these huge functions but we need to use bits and pieces of them in the development of the new product.

Now I have to spend days and possibly weeks refactoring dozens of functions and breaking them apart in to managable services so we can not only use them, but also extend and test them.

I'm afraid what Carmack was talking about was meant to be taken with a grain of salt and not applied as a "General Rule" but people will anyway after reading it.

[akkartik]

Perhaps it suggests our way of testing needs to change? A while back I wrote a post describing some experiences using white-box rather than black-box testing: http://web.archive.org/web/20140404001537/http://akkartik.na... [1]. Rather than call a function with some inputs and check the output, I'd call a function and check the log it emitted. The advantage I discovered was that it let me write fine-grained unit tests without having to make lots of different fine-grained function calls in my tests (they could all call the same top-level function), making the code easier to radically refactor. No need to change a bunch of tests every time I modify a function's signature.

This approach of raising the bar for introducing functions might do well with my "trace tests". I'm going to try it.

[1] Sorry, I've temporarily turned off my site while we wait for clarity on shellsock.

[thisGuysAccount]

Something to consider, and this is only coming off the top of my head, is introducing test points that hook into a singleton.

You're getting more coupling to a codebase-wide object then, which goes against some principles, but it allows testing by doing things like

function awesomeStuff(almostAwesome) {

  MoreAwesomeT f1(somethingAlmostAwesome) {
    TestSingleton.emit(somethingAlmostAwesome);
    var thing = makeMoreAwesome(somethingAlmostAwesome) 
      // makeMoreAwesome is actually 13 lines of code,
      // not a single function
    TestSingleton.emit(thing);
    return thing;
  };

  AwesomeResult f2(almostAwesomeThing) {
    TestSingleton.emit(almostAwesomeThing);
    var at = makeAwesome(awesomeThing); 
      // this is another 8 lines of code. 
      // It takes 21 lines of code to make somethin 
      // almostAwesome into something Awesome, 
      //and another 4 lines to test it.
      // then some tests in a testing framework
      // to verify that the emissions are what we expect.
    TestSingleton.emit(at);
    return at;
  }

  return f2(f1(almostAwesome));

}

in production, you could drop testsingleton. In dev, have it test everything as a unit test. In QA, have it log everything. Everything outside of TestSingleton could be mocked and stubbed in the same way, providing control over the boundaries of the unit in the same way we're using now.

[robrenaud]

How brittle are those tests though?

I've had to change an implementation that was tested with the moral equivalent to log statements, and it was pretty miserable. The tests were strongly tied to implementation details. When I preserved the real semantics of the function as far as the outside system cared, the tests broke and it was hard to understand why. Obviously when you break a test you really need to be sure that the test was kind of wrong and this was pretty burdensome.

[akkartik]

I tried to address that in the post, but it's easy to miss and not very clear:

"..trace tests should verify domain-specific knowledge rather than implementation details.."

--

More generally, I would argue that there's always a tension in designing tests, you have to make them brittle to something. When we write lots of unit tests they're brittle to the precise function boundaries we happen to decompose the program into. As a result we tend to not move the boundaries around too much once our programs are written, rationalizing that they're not implementation details. My goal was explicitly to make it easy to reorganize the code, because in my experience no large codebase has ever gotten the boundaries right on the first try.

[RangerScience]

I've dealt with similar situations, and it was what led to me to favor the many-small-functions myself. I like this article because by going into the details that convinced him, Jon Carmack explains when to take his advice, not just what to take his advice on.

I think maybe the answer is that you want to do the development all piecemeal, so you can test each individual bit in isolation, and /then/ inline everything...

That sound like it might be effective?

[nakor]

I'm not sure. If you then go head and inline the code after, your unit tests will be worthless. I mean it could work if you are writing a product that will be delivered and never need to be modified significantly again (how often does that happen?). Then one of us has to go and undo the in-lining and reproduce the work :)

[RangerScience]

I think I'm going to say that, if it's appropriately and rigorously tested during development... testing the god-functionality of it should be OK.

Current experience indicates however that such end-product testing gives you no real advantage to finding out where the problem is occurring, since yeah, you can only test the whole thing at once.

But the sort-of shape in my head is that the god-function is only hard to test (after development) if it is insufficiently functional; aka, if there's too much state manipulation inside of it.

Edit: Ah, hmm, I think my statements are still useful, but yeah, they really don't help with the problem of TDD / subsequent development.

[Chris_Newton]

Current experience indicates however that such end-product testing gives you no real advantage to finding out where the problem is occurring, since yeah, you can only test the whole thing at once.

I’m not so sure. I’ve worked on some projects with that kind of test strategy and been impressed by how well it can work in practice.

This is partly because introducing a bug rarely breaks only one test. Usually, it breaks a set of related tests all at once, and often you can quickly identify the common factor.

The results don’t conveniently point you to the exact function that is broken, which is a disadvantage over lower level unit tests. However, I found that was not as significant a problem in reality as it might appear to be, for two reasons.

Firstly, the next thing you’re going to do is probably to use source control to check what changed recently in the area you’ve identified. Surprisingly often that immediately reveals the exact code that introduced a regression.

Secondly, but not unrelated in practice, high level functional testing doesn’t require you to adapt your coding style to accommodate testing as much as low level unit testing does. When your code is organised around doing its job and you aren’t forced to keep everything very loosely coupled just to support testing, it can be easier to walk through it (possibly in a debugger running a test that you know fails) to explore the problem.

[6d0debc071]

> I'm not sure. If you then go head and inline the code after, your unit tests will be worthless.

Local function bindings declared inline perhaps? It seems to me you could test at that border.

[collyw]

Could this not be achieved in an IDE - "inline mode". It could display function calls as inline code and give the advantages of both.

[corysama]

If it's done strictly in the style that I've shown above then refactoring the blocks into separate functions should be a matter of "cut, paste, add function boilerplate". The only tricky part is reconstructing the function parameters. That's one of the reasons I like this style. The inline blocks often do get factored out later. So, setting them up to be easy to extract is a guilt-free way of putting off extracting them until it really is clearly necessary.

But, it sounds like what you are dealing with is not inline blocks of separable functionality. Sounds like a bunch of good-old, giant, messy functions.

[akkartik]

I think the claim is that if you don't start out writing the functions you don't start out writing the tests, and so your tests are doomed to fall behind right from the outset.

I'm not fanatical about TDD, but in my experience the trajectory of a design changes hugely based on whether or not it had tests from the start.

(I loved your comment above. Just adding some food for my own thought.)

[sheepmullet]

"I'm not fanatical about TDD, but in my experience the trajectory of a design changes hugely based on whether or not it had tests from the start."

I'm still not sold on the benefits of fine grained unit tests as compared to having more, and better, functional tests.

If the OPs 1k+ methods had a few hundred functional tests then it should be a fairly simple matter to re-factor.

In "the old days" when I wrote code from a functional spec the spec had a list of functional tests. It was usually pretty straightforward to take that list and automate it.

[akkartik]

Yeah, that's fair. The benefits of unit tests for me were always that they forced me to decompose the problem into testable/side-effect-free functions. But this thread is about questioning the value of that in the first place.

Just so long as the outer function is testable and side-effect-free.

[mercurial]

Say you have a system with components A and B. Functional tests let you have confidence that A works fine with B. The day you need to ensure A works with C, this confidence flies out of the window, because it's perfectly possible that functional tests pass solely because of a bug in B. It's not such a big issue if the surface of A and C is small, but writing comprehensive functional tests for a large, complex system can be daunting.

[mark-r]

The intro to the post has Carmack saying he now prefers to write code in a more functional style. That's exactly the side-effect-free paradigm you're looking for.

[_delirium]

Even most of the older post is focused on side-effecting functions. His main concern with the previous approach is that functions relied on outside-the-function context (global or quasi-global state is extremely common in game engines), and a huge source of bugs was that they would be called in a slightly different context than they expected. When functions depend so brittly on reading or even mutating outside state, I can see the advantage to the inline approach, where it's very carefully tracked what is done in which sequence, and what data it reads/changes, instead of allowing any of that to happen in a more "hidden" way in nests of function-call chains. If a function is pure, on the other hand, this kind of thing isn't a concern.

[lutusp]

> They're written in the style that Carmack describes, and I have methods that span more than 1k lines of code.

I don't think that's the kind of "inlining" being discussed -- to me that's the sign of a program that was transferred from BASIC or COBOL into a more modern language, but without any refactoring or even a grasp if its operation.

I think the similarity between inlining for speed, and inlining to avoid thinking very hard, is more a qualititive than a quantitative distinction.

[dllthomas]

"I think the similarity between inlining for speed, and inlining to avoid thinking very hard, is more a qualititive than a quantitative distinction."

I think what's being discussed here is quite either of those - this seems to be "inlining for visibility" and possibly "inlining for simplicity".

[dllthomas]

Is not quite either of those.

[GFK_of_xmaspast]

Have you seriously never written a 1000 line routine in C from scratch?

[lutusp]

Sure, before I knew how to write maintainable code. Before I cared to understand my own code months later.

My first best-seller was Apple Writer (1979) (http://en.wikipedia.org/wiki/Apple_Writer), written in assembly language. Even then I tried to create structure and namespaces where none existed, with modest success.

[mdda]

Another great comeback for the annals of HN (like https://news.ycombinator.com/item?id=35083)

[bsdetector]

Maybe you should just be testing the 1k functions, if that even, and not the individual steps they take. The usefulness of testing decreases with the size of the part being tested, because errors propagate. An error in add() is going to affect the overall results, so testing add() is redundant with testing the overall results and you are just doing busywork making tests for it.