|
|
|
|
|
|
by runeks
3271 days ago
|
|
|
The solution to this is to deliver arguments at runtime, rather than baking them into the program as constants. Describe some computation by a data structure that is delivered at runtime, and see which implementation does best. That way there can be no cheating. |
|
|
Functions already take their arguments at runtime. Except when they don't, due to optimisation.
For a benchmark to be automated, reproducible, etc. those constants have to be baked in somewhere, even if it's in a Haskell program using FFI (as in the article), or a shell script, etc. Whilst optimisers don't (yet) cross the language/process boundary, it still makes sense to include such sanity checks, rather than assuming we know what the optimiser will/won't do.
After all, the whole point of a benchmark is to gather evidence to question/ground the assumptions of our mental model. The less we assume, the better. The more evidence we gather, the better.