[scott_s]

In academic systems papers, every performance claim needs to be backed up by an experiment. But you can get credit for features even if you argue that it is possible to implement that feature with your design, even if you didn't actually do it.

In production software, this is flipped. Every feature claim needs to have an associated test, as it's a contract with your user. But when it comes to performance, everyone just waves their hands.

I'm being a little glib. But production software has to work. You'll spend far more time dealing with all of the "less interesting" details and edge cases than with research software. As ams6110 points out, this means more focus on testing, maintenance and good design. But I do want to emphasize testing - sometimes you'll spend more time testing something than actually implementing it. There's also often many more residual effects from dependencies elsewhere in the ecosystem you're working in. That's not typical in academic software.

[pacaro]

I've encountered academic code that implements the state of art, most efficient, algorithm for solving some problem.

The code that comes to mind had the following properties: over 20 years old; written in C and badly converted to C++ somewhere along the way (the stuff-all-the-globals-into-a-class approach); a combinatorial explosion of #define and #ifdef statements (covering all the experiments in the original paper)

In the paper, it is clear that one of the experiments wins, and why. So...

Step 1: remove all dead code.

Step 2: observe that the algorithm needs no dynamic memory allocation, remove all but 1 call to malloc, calloc, realloc, and free.

Step 3: the use of float can be replaced by correctly scaled 64-bit unsigned integers, with no loss of precision

Step 4: rewrite entirely in modern C++, this has two benefits, a) I get to use the <algorithms> library (judiciously, this simplifies the code enormously), and b) the code can send clearer messages to the compiler than the mid-90s liberal sprinkling of the 'register' keyword.

The net result is no asymptotic improvement whatsoever — arguably a slight improvement for very large N as heap performance starts to interfere, but nothing worth the effort.

However, the code now has tests (step 0), is clean and maintainable, is 10% of the size, and is 5-30x faster (depending on the shape of the data)

[hueving]

This is not a good rule of thumb, it depends on what your research is. In most cases I've dealt with (security) the academic software displays terrible performance characteristics and is very buggy. The industry application that surfaces years later does not have these problems but it doesn't present anything novel.

[scott_s]

I said academic systems papers; their evaluation criteria is generally performance.

[hueving]

Not really. Their evaluation criteria is usually only performance if there are existing established ways of doing something.

[scott_s]

In my academic systems papers, the evaluation criteria is generally performance. I have to admit I'm not quite sure what you mean.