[igouy]

> If you're asking what multi-lingual benchmark suites offer good coverage - I don't know.

In which case, given: The target audience wonder "Which programming language is fastest?" :there doesn't seem to be support for your claim that: "Having more domain coverage is easier and more valuable…".

> Is it better than nothing? Maybe, but not by much…

The benchmarks game: provisional and modest.

[pron]

> There doesn't seem to be support for your claim that: "Having more domain coverage is easier and more valuable…".

Since programming languages specifically optimise differently for the different conditions I listed above, the "support" for my "claim" is that it's obviously true. No one who implements languages or runtimes will dispute it.

But I don't understand the logical implication. The existence or nonexistence of good information says nothing about the value of the information we do have. If all you know is how much cash I have in my wallet, the fact that no one has ever published how much money I have in my bank account doesn't make the information you have more relevant as an estimate of my wealth. That information is irrelevant regardless of whether or not you have access to the relevant information. That information being available is not what's needed to "support" my "claim" that what you know is irrelevant. All you need to know is how people keep their money.

> The benchmarks game: provisional and modest.

I would say it's more like a website comparing US presidential candidates through polls only in Alabama. A more appropriate description than "provisional and modest" would be that it doesn't actually give us valuable information about the candidates' chances.

If people know how US elections work, such information could be put in context, but I don't know how many programmers understand how languages and runtimes optimise performance. Merely saying it's partial/provisional/modest is insufficient to give people the appropriate context.

[igouy]

Sorry, I'm just not interested in wading through your analogies.

[pron]

My point is that the website offers a very low coverage of the codepaths that affect language performance, and all it says is that the coverage is incomplete without saying just how incomplete it is. People who don't know that, for example, optimisation in some languages can vary greatly depending on whether there's one compilation unit or several (especially since most/all examples have one compilation unit while virtually all programs have many) and that memory management depends heavily on the variety of object sizes and may degrade over time (especially since the examples all run for a short time and have a low variety of object lifetime and size while most real programs are different), can come to very wrong conclusions.

Sorry for yet another analogy, but it's just like me telling my boss that my test suite passes but it's incomplete without telling him that I've tested only 20% of the program's functionality.

The website should at least explain just how skewed the results are in favour of conditions that arise in small, short-lived programs without (competitive) concurrency and that larger and/or longer-lived and/or concurrent programs can exhibit very different behaviour. These conditions aren't a small matter. They're among the primary motivations for the huge investment over the last few decades in moving GCs and JIT compilers.

[igouy]

> People who don't know … can come to very wrong conclusions.

Always.

> … website should at least explain…

Whatever the explanation, it would never be sufficient for you.

As you said -- "If you're asking what multi-lingual benchmark suites offer good coverage - I don't know."

[pron]

A website that doesn't give people the relevant context for the data it offers is not a good website, especially when the data cannot be understood without that context and the context is not at all well known. That people may misinterpret the information if it were given to them doesn't change that.

> Whatever the explanation, it would never be sufficient for you.

Since I work on compilers and runtimes I know just how small the coverage offered on that website is. For me, no benchmark suite may ever be enough. Still, offering some of the information that is needed to put the results into context would be better than offering none.

Again, I'm not saying that the website is intentionally misleading and I have nothing against the people behind it. They may have a very good reason for why it's of such low quality. I'm just pointing out that it is.