[j-pb]

There's only one way to solve this: Simplicity.

Ironically this is also what occams razor would demand from good Science, so you'd have a win win scenario, where you both create good software and good research, because you focus on the simplest most minimal approach that could possibly work.

[gmueckl]

How do you keep a codebase simple when you need have things in it like implementations of state of the art algorithms to compare against and the previous iterations of your own method so that you can test whether you're actually improving? Then, depending on what you're doing, there's also all the extra nontrivial code for tests and sanity checks of all these implementations.

Simplicity is a nice dream. The realities of research are very often stacked against it.

[j-pb]

How the heck to you hope to gain any insighfull metrics when you've got a cobbled together mess that you only half understand. For what it's worth you might only be benchmarking random code layout fluctuations.

I've seen research groups drown in their legacy code base.

The issue of juggling too many balls you describe is one you only have to begin with because the state of the art implementations are so shoddy to begin with.

Research suffers as much as everybody else from feature creep. Good experiments keep the number of new variables low.

[gmueckl]

Research code is not only written to measure runtime. Reducing the argument to only that aspect is not helping the discussion.

And you say it yourself: good experiments change a single variable at a time. So how do you check that a series of potential improvements that you are making is sound?

[childintime]

It seems Julia has the answer: https://arstechnica.com/science/2020/10/the-unreasonable-eff...

[gmueckl]

I can't quite follow what the article is trying to describe because of the heavy use of analogies.

A Google search makes it look like Julia has a mechanism where you can extent the sets of overloads of a function or method outside the original module. The terminology is different (functions have methods instead of overloads in their speak). I don't see how that feature solves the problem in practice.

[cdsousa]

Julia multiple dispatch allows dynamic polymorphism.

Check https://stackoverflow.com/questions/1801216/what-is-the-diff... and https://www.youtube.com/watch?v=kc9HwsxE1OY

[WanderPanda]

In my experience simplicity and generality don‘t go well with performance. If you want to build something that can be used for all kinds of problems and it is simple it will be slow as hell compared to the (dirty) optimised code running hardcoded structures on the GPU

[j-pb]

Simplicity pretty much excludes generality in a lot of cases, you're only able to port code to the GPU if it wasn't a million LOC to begin with, so you're pretty much making the case for it.

Note that Simple != Easy or Naive

Hardcoded structures is potentially exactly the kind of simplicity needed.

What's not simple is a general "this solves everything and beyond" code-base with every imaginable feature and legacy capability.