[gammadens]

I agree, but I also think a comprehensive set of standard benchmarks can be useful to get an overall sense of how language instantiations perform and how things change. It's fairly clear that in general some language implementations are much slower across the board than others, even if for many other comparisons the distinctions are fine or depend on domain.

My overall sense is that there's been a pull back from general benchmarking compared to say, 15 years ago, and it's unfortunate, because it leaves the benchmarking to developers of languages, compilers, and whatnot. This provides an opportunity to show of the best-case scenarios for the languages, but also for them to hide the areas of weakness -- and those hidden areas are often the mine traps for those deciding whether to invest resources in a new language.

Having a standard, comprehensive set of problems helps address this "hiding." I also think there's value in naive benchmark programs as well as "expert" tuned ones: not everyone is going to optimize every single scenario in every language.

The one thing I've never seen implemented well is some measure of "ergonomics" or "high-level" versus "low-level" aspects of a language, which also seems important to me. Some of that is going to be subjective but some of it not.