[bhauer]

To your last point, Terretta, my colleagues and I have had lengthy (but unfortunately inconclusive) discussions about how to represent the efficiency dimension that we expect many readers would like to visualize.

The easier dimension--performance--became our first goal and the source code in Github is expected to very loosely fill the role of answering questions of efficiency. But we know that's a barely serviceable solution to the challenge, and that is especially true as pull requests increase the number of frameworks.

The challenge of representing efficiency succinctly remains.

We have considered lines of code and I think that for all its weaknesses, that is the best proxy for efficiency that I am aware of. Nevertheless, I'd like to contend with some of the following specific issues:

1. Many frameworks create boilerplate when you create a new application. Do we count those LOC?

2. Many frameworks on dynamic languages copy their entire corpus of functionality as source files into the application's root. Certainly we don't count those. Check out our Github repo's colored language bar at the top right. :)

3. Do build scripts count as LOC?

4. Do configuration files count as LOC?

Ultimately, I retain (irrational?) fear of contributing to enshrining LOC as a metric because unlike performance--where higher is always definitively better--lower LOC is not always definitively better.

I was asked separately if we had any data about how long it took us to build tests in the various frameworks. We didn't take detailed logs, but we do have rough numbers. Nevertheless, I've been reluctant to share how long it took us to implement the test code because it's a biased sample. Our previous experiences make us well disposed to some platforms and languages while we make silly time-consuming mistakes on others.

[kbenson]

This is how I would answer these questions:

> 1. Many frameworks create boilerplate when you create a new application. Do we count those LOC?

No, as long as the lines did need to be changed. Any line you must alter (not just add code before/after) should be considered a line of code you had to write to get to a functioning implementation. You had to understand the line before altering, so could have written it entirely yourself.

As such, diffing against a reference boilerplate file is a good indication of lines required.

> 2. Many frameworks on dynamic languages copy their entire corpus of functionality as source files into the application's root. Certainly we don't count those. Check out our Github repo's colored language bar at the top right. :)

Take whatever you are given as a boilerplate starting point for a new project, and diff the final implementation tree against that. Special care may need to be given to implementation that default to installing third party libs into the local framework lib path (if any exist), so those are not counted.

> 3. Do build scripts count as LOC?

No, I would think not.

> 4. Do configuration files count as LOC?

This is a bit more complicated, but for simplicity's sake it may be easiest to treat it just like any other reference file as in #1 and #2.

> Ultimately, I retain (irrational?) fear of contributing to enshrining LOC as a metric because unlike performance--where higher is always definitively better--lower LOC is not always definitively better.

I agree, but without something better, it's what we have.

I do think there's ways we could get to a more useful metric using relative lines of code required to implement something between languages, and then compare relative LOC to host language, but that's way outside the scope of this benchmark, and requires information we don't have (to my satisfaction) yet.

[bhauer]

Thanks for the thoughts here, kbenson.

I like the idea of using a diff be judge the total LOC. That reminds me that before we did the initial commit to Github, we had briefly entertained doing more or less that: we were going to commit the initial boilerplate for each framework as commit #1 and then the resulting tests as commit #2. To save on effort, we ultimately did not do that, but it is possible for us to go back to our original local Git repo to glean that information.

Using that approach, I think even build scripts could be included--that is, if you need to modify the build script and it shows up in a diff, then that line counts.

[Terretta]

1. Many frameworks create boilerplate when you create a new application. Do we count those LOC?

Yes if it's code someone could edit. Someone unfamiliar with the framework needs to read these lines of code. Boilerplate may be "pre-typed" for you, but it's part of "your" app, and it's optional. That's not the same as a lib or the framework itself. You could start your app differently.

The new hire has to deal with these LOC regardless of who typed them (you or a wizard), so they're significant.

2. Many frameworks on dynamic languages copy their entire corpus of functionality as source files into the application's root. Certainly we don't count those. Check out our Github repo's colored language bar at the top right.

Agree this should not be counted. Only what's part of the app in question. Put another way, if those directories could be "hidden" from the new hire, and he could do his job, don't count them.

3. Do build scripts count as LOC?

No.

4. Do configuration files count as LOC?

Yes. The Spring framework's pom.xml or any dependency injection "configuration" file is crucial, so yes. On configs, though, I'd guess kbenson's idea about diff is reasonable.

- - -

However, about diff on #1 above, I'm concerned that with diff you're really just measuring how different the test is from the boilerplate. That says more about the test than about the framework. Someone could also optimize their framework's pre-typed boilerplate to match your test. (As we see on browser rendering benchmarks, for example.)

Really, if it's code that's typed (by the framework or by you) into an application, it should be counted.

I'm coming to this from the standpoint of hiring a new developer onto a project. More time is spent maintaining projects than jumpstarting them. At some point really soon after jumpstarting the app, boilerplate and edited code are going to be intermingled and indistinguishable. The new guy has to wrap his head around it all. So I would like any code typed by you or for you to be counted.

And beyond LOC, it matters how those lines are distributed. If you have to open five nested files to find out what one hello world is doing, even if each file only has a single 3 line function in it, that's complexity that matters. Number of files and number of directories both make a framework feel very different.

Most benchmarks are for weekend MVPers. Your benchmarks are moving into "real world" territory. In the real world, being able to wrap your head around someone else's existing project matters. LOC, files, directories, and dependencies, factor into that heavily.

[kbenson]

> The new hire has to deal with these LOC regardless of who typed them (you or a wizard), so they're significant.

That's true. Maybe a two numbers, one with and one without boilerplate? That way someone could get an idea of what they are looking at. (I know, I'm just stacking more and more work up...)