[etiene]

If your intention is not to make a good comparison, then sure. I've seen other papers on language implementation and benchmarks, and researchers were way more careful than this, individually inspecting the algorithms they were using.

I happened to spot a problem on this one on the Lua part and also on some other parts concerning how they classify things, because it happens to be things I know more. I'm not going out of my way to look for problems in the rest of the paper because I'm not a reviewer and I have better things to do.

But if I was the one way conducting this academic research, then I would want the results not to be bogus. The difference here is this is not a blog post about some bullshit comparison someone is making. If that was the case, then alright, if public contributed algorithms failed to represent exactly what was being looked for, why not go for whatever is out there. I just expect a more rigorous procedure from academics.

[igouy]

> I just expect a more rigorous procedure from academics

Are they likely to be expert in all those programming languages?

----

If Lua wasn't included would you complain ?

( Like this:

https://news.ycombinator.com/item?id=15255427

https://news.ycombinator.com/item?id=15251242

https://news.ycombinator.com/item?id=15251144

)

[etiene]

> Are they likely to be expert in all those programming languages?

No, but again, I take paper results very seriously. Misleading results are bad for science, period. Here is an example of more rigorous research: https://arxiv.org/pdf/1602.00602.pdf

> If Lua wasn't included would you complain ?

Maybe I would, who knows. I did also mention in another comment it would be nice if Julia was there. It depends how relevant they are for the research being conducted. As a language used very often in microcontrollers, Lua is very relevant for energy efficiency research. I don't know anything about Forth, but I wouldn't just dismiss people saying it should be there with "then contribute forth algorithms to the benchmarks game yourself".

[igouy]

> … I take paper results very seriously…

"disgusts me" & "pure dishonesty" don't seem like a serious response.

> … an example of more rigorous research…

Which again uses programs that were contributed to the benchmarks game.

> … wouldn't just dismiss people…

I wouldn't just dismiss them: I'd tell them that others have every right to present what seems important to them, and exclude what seems less important -- without being accused of dishonesty.

There's only so-much time & money.