| This is how I would answer these questions: > 1. Many frameworks create boilerplate when you create a new application. Do we count those LOC? No, as long as the lines did need to be changed. Any line you must alter (not just add code before/after) should be considered a line of code you had to write to get to a functioning implementation. You had to understand the line before altering, so could have written it entirely yourself. As such, diffing against a reference boilerplate file is a good indication of lines required. > 2. Many frameworks on dynamic languages copy their entire corpus of functionality as source files into the application's root. Certainly we don't count those. Check out our Github repo's colored language bar at the top right. :) Take whatever you are given as a boilerplate starting point for a new project, and diff the final implementation tree against that. Special care may need to be given to implementation that default to installing third party libs into the local framework lib path (if any exist), so those are not counted. > 3. Do build scripts count as LOC? No, I would think not. > 4. Do configuration files count as LOC? This is a bit more complicated, but for simplicity's sake it may be easiest to treat it just like any other reference file as in #1 and #2. > Ultimately, I retain (irrational?) fear of contributing to enshrining LOC as a metric because unlike performance--where higher is always definitively better--lower LOC is not always definitively better. I agree, but without something better, it's what we have. I do think there's ways we could get to a more useful metric using relative lines of code required to implement something between languages, and then compare relative LOC to host language, but that's way outside the scope of this benchmark, and requires information we don't have (to my satisfaction) yet. |
I like the idea of using a diff be judge the total LOC. That reminds me that before we did the initial commit to Github, we had briefly entertained doing more or less that: we were going to commit the initial boilerplate for each framework as commit #1 and then the resulting tests as commit #2. To save on effort, we ultimately did not do that, but it is possible for us to go back to our original local Git repo to glean that information.
Using that approach, I think even build scripts could be included--that is, if you need to modify the build script and it shows up in a diff, then that line counts.