Years ago i was naive enough to believe these benchmarks.
Now i know these are Potemkin-village level stunts and political maneuvering by language activists and staff of that site.
If you really want actual benchmarking you have to do it yourself and read the .asm output for comparison. Another caveat is that some languages are genuinely faster in one hyperoptimized case(even faster than C) but fail horribly when things go more complicated and abstraction creep makes them much slower in most cases.
The problem is subjective and thus more difficult to provide evidence for.
I don't think the parent doubts the runtimes or is claiming that the code for his pet language wasn't written properly and therefore the site is a sham. The issue is: what is reasonable code?
Benchmarks invariably become a game of massaging the code to produce the optimal assembly. And that's the problem. The reason that we invented higher level languages was so we could stop thinking about assembly. The point of most higher level languages is not to optimize CPU cycles, but to optimize human brain cycles. If we only cared about speed (and not at all about development time), we would just write all code in assembly.
I am not particularly interested in how fast meticulously fine tuned code in a given language runs. I have no doubt that, given enough time, an excellent programmer can finely craft a block of code to run very fast in any half decent language. And this site is evidence of that.
What I want to know, and what benchmarks never show, is which languages make writing performant code simple, idiomatic, and natural. Which languages give me good performance, without having to really work for it?
This is an amazing site, but it is a stretch to think that it compares languages. It compares programmers and their implementations. The implementations are more like their own art form, like Chess or something.
>> I don't think the parent doubts the runtimes or is claiming that the code for his pet language wasn't written properly and therefore the site is a sham. <<
We could always take the parent at their word -- "Now i know these are Potemkin-village level stunts and political maneuvering by language activists and staff of that site.".
Have you looked at the winning submissions code? Every single one I saw (not all of them) was just a straightforward implementation of the algorithm / solution. I don't see much weirdness.
Yes. For example I noticed the typescript one. The implementation is straightforward, yes, but it is just plain javascript. It doesn't need TSC at all. So why call it typescript? If you're going to say it is typescript at least use types, no?
Or the ruby ones. Not much like ruby I have ever seen. It's not particularly difficult to write C code that can be easily called from ruby which is why I guess most ruby programmers wouldn't bother to write code like this in ruby. If you want C, just write C!
I think all the implementations are straightforward as implementations. But few are the most idiomatic way to express the alogorithm in that language.
I had the same observation. I'm a Perl fan, but I can see some submissions were solely motivated to make Perl appear competitive. Like using threads in Perl, which isn't often done in real life.
I'm not sure how much evidence is needed. It seems obvious that people who are fans of language X are going to submit code with the sole intention of bumping up performance at any cost.
The site is still interesting, but probably useless for real world decision making.
Doesn't matter, if you don't use threads in Perl when you could, that's your fault and benchmarks like this serve to remind you should. There are folks who use threads with their Perl code in real life btw
I don't find the .asm output to be meaningful at all. The realistic comparison for me is if I write a solution the easiest/most convenient way in each language, which language is naturally more efficient.
Hey, the benchmark game got a new website (at some point in the several years since I looked at it last)! Almost simplistic, but nice to read.
As someone that works predominantly in a language long considered passé (Perl), I always kinda relish seeing that it is still faster than Python and Ruby for many problems. As Python and Ruby have gotten faster, I'd sort of assumed they were both notably and clearly faster than Perl, by now, but that's not the case though the difference is now quite small and probably debatable.
Python was never really designed for speed. It was designed for simplicity and clarity of behaviour. The beauty of the age we live in is that we have such a broad array of tools to apply to different problems. Performance is often not the problem that needs solving, but when it is, you've got many options too!
I'm not disparaging Python (or Ruby). Merely taking small pleasures where I find them. I work in Perl because of a huge array of existing code in Perl, not because I hate Python or Ruby (I've built stuff in Python and Ruby, too, and like them both).
Perl is only faster here in nbody because I tricked it. I used a tricky compile-time optimization for faster array index access. You wouldn't do that in the real world, and in the real world PHP, ruby and python are faster. cperl instead is only beaten by PHP. But the benchmark game is not open for faster alternate engines like luajit, nim, crystal, pypy, truffle-graal, ... or even slower ones like Perl6, which limits the impact of those better engines.
Decades ago it was open and written in Perl. Then it was rewritten in Python and closed down. Go figure.
>> a tricky compile-time optimization for faster array index access. <<
And you even comment on that usage in your code.
>> Decades ago it was open and written in Perl. Then it was rewritten in Python and closed down. Go figure. <<
At best your comment is disingenuous.
You can download the Python measurement scripts, use them with luajit, nim, crystal, pypy, truffle-graal or Perl6 programs and publish the measurements.
Perhaps this is merely a reflection of the difficulty of finding where to download the measurement scripts? I found them (here: http://benchmarksgame.alioth.debian.org/play.html ) but, it took some clicking. Maybe a github link on the front page, or similar, would alleviate some of the frustration the previous poster had about it.
I assumed the sources were available, somewhere, but until I went looking for it, I wouldn't have been able to say so with confidence. That's not an obligation, of course...volunteers should feel free to do whatever they want with their projects. But, if you wanted to keep the community involvement the game had historically, linking source in the lingua franca of the day (github or gitlab, or whatever, seems pretty standard today) would go a long way.
If course I have the benchmarks, because that's how everybody benchmarks it. But rather with the old perl scripts, not with your new python scripts. They suck. Before it was trivial to bench, but not trivial to setup all the environments.
The problem is not that someone else can do it, the problem is YOU need to do it. You got the name reserved for it, you got the machines to do it, you decide to rewrite it and throw them out, no one is looking somewhere else for better and more benchmarks.
The big competitors with better and faster engines have no incentive to match performance when their name is not on the list, and they just publish their own results. Users don't look somewhere else.
>> Before it was trivial to bench, but not trivial to setup all the environments. <<
That's what it's like now -- trivial but time consuming.
There were pain-points with the old perl scripts (and nested make files) but I would have continued to put-up with the problems and would have continued to use the old perl scripts.
But (back in 2008) I couldn't see how to set-affinity with Perl and I could see how to do that with Python.
That's why I rewrote the measurement scripts in Python.
Dude. What's up with attacking someone who does things voluntarily (I presume) for not doing them the way you want them to? That's pretty damned presumptuous.
Open Source stuff really only ever has one guarantee: If it breaks, you get to keep the pieces.
You can put it back together any way you want, but you can't demand they give you a new one that meets your expectations. Unless you want to pay them to do so, and they're willing to accept the money and terms.
Are you the maintainer? Thanks for keeping it running all these years, if so.
I don't go there often, but it's nice to know there's a way to get a vague understanding of relative performance for languages for those occasions when one needs to know "is it fast enough?"
My somewhat glib comment was meant to point out that even had PHP been very fast back when PHP hate was a popular past time it would still have been hated for other reasons.
Ruby was (very) slow back then (notably and measurably slower than PHP, Python, or Perl) and people still loved it. The warts on PHP that got the most bad press were rarely related to performance.
So, yes PHP used to be slower than it is today, by a lot. But, speed wouldn't have immunized it against the criticism leveled against it. All the effort going into fixing performance for PHP coincided with fixing some of the bigger warts of the language, as well...so much of the hate for PHP has dwindled in recent years.
The fact that people building for the web are much less often required to work in PHP also reduces the hate.
It was low performance, but not slow, its simple model meant it was performing enough for the task (DB IO was the bottleneck most of the time).
Since PHP5, the lead team made progress into making it solid. And in PHP7 some eastern/russian guy hacked the VM a lot. Adding new traits and improving memory usage and performance a lot too.
The "hatred" comes from old days (v3,v4):
- massive inconsistency in naming
- global side effect was a norm
- bugs, bugs, bugs
Combined with the nascent web boom, tons of new kids (that includes me) started having fun with html/php. A large mass of the web was written with it, carrying all the bad design decisions in forms of security bugs and impossibly bad code.
ps: I audited one famous wordpress plugin and it was 10 pages long of redundant code in 3 big loops, dead code everywhere, commented code released in production. All this to add a few links here and there.
It's not necessarily a performance issue, unless you're Facebook (traffic wise) you might never worry about that, but it's more about the developers who use PHP[0] it's a mixed bag of below junior level to senior level developers. It's hard to know which type of codebase you will get your hands on until you do. There's other issues I personally have with it, though I use it if it's not code I can easily reinvent without investing way too much of my own time on it e.g. a forum or a blog engine.
"unless you're Facebook (traffic wise) you might never worry about that"
Facebook hasn't spent a lot of time worrying about their choice of PHP either. Facebook's primary language is Hack, a mostly compatible derivative of PHP. Prior to Hack (~2014) Facebook used traditional PHP, which they reimplemented a couple times for performance reasons. It seems that Facebook has spent its time worrying about how NOT to leave PHP.
Makes one wonder about the value of this daily new programming language era we're in at the moment.
> I thought everyone hating PHP ( also) because it was slow
Most of the replies here are stating that PHP is fast. It can be for these kind of benchmarks.
But, it does have the reputation you're suggesting. That reputation, though, is based on the old CGI model. Most php apps are built in a way that all of the initiation and setup code is run for every single request. That makes it comparatively slow.
Some PHP apps get around that by using a shared memory cache, like apcu, to box around that. That helps a lot, but isn't the same as a typical golang or node.js app where startup code is truly run once, and new requests go through a much shorter path.
All of that, though, is less a "language" thing and more of a runtime thing. You can, for example, use Facebook's HHVM+Proxygen and run PHP code in the exact same way as node.js.
php7 was threatened by hhvm and was completely rewritten by keeping backcompat. It's now the fastest major old scripting language. js and luajit are in the very fast ballpark, php in the middle and the other 3 far behind.
PHP also has the best language features and IDE integrations, the only remaining problem is the legacy stdlib and devbase. Ruby with an inliner and method cache would catch up, but I don't see it happen. Python is doomed by its architecture. Perl has a faster interpreter loop, but data structure bloat and similar dev problems as the other 2.
pypy changed not only the vm, but also the internal data structures. It's totally different.
pypy's problem is that is only optimizes loops and often called functions, not the general interpretation, as e.g. PHP7 or lua did.
For normal, simple small scripts it's not that big a benefit, and you loose all the extensions, (though that's being worked on).
That has been less true for the last few years: C extensions are still a concern, although many of the major ones work now, but I've had a significant number of programs run faster simply by using pypy. It's a full JIT compiler and one nice area that improves is regex handling so all of my log-related work got faster with zero effort.
The means that a lot of rather bad (inexperienced?) programmers use it, which means there is a lot of bad PHP code out there.
People see that bad code and assume the language is bad.
The language has some warts and inconsistencies, and people are eager to point them out as justification for their hate. But for real programming most of them hardly matter.
this is something a bit naive - basically all dynamic languages have the same low barrier and can produce the same crappy code, see for example javascript or ruby (with RoR). Python too, considering the fact that it's used by a lot of people that their main job is not software engineering.
The biggest difference between PHP and the other dynamic languages, though, is that PHP came out with pretty popular stuff like wordpress, joomla, magento or drupal that they all share crappy code coz they all have a large codebase started before PHP managed to fix some stuff starting from version 5.3
That comparison is complicated: it's definitely easy to drop a .php file on a server but once you start talking about frameworks, using a newer version than your OS distribution shipped, adding C extensions, etc. it's easy to find counter-examples, and the Go proponents are not wrong to observe that deploying a single file is easier than either.
This is also becoming less of a distinguishing factor in the container era where the answer for a number of cases is “Extend a base Docker image”.
At least PHP offers that option at all, and it's well-supported and well-documented. It's also very easy. Good luck doing the same with any other language.
If you really, really try it's possible in Python, C# and Java too, but ... wtf.
As slow as it might be to develop something from scratch in c++, perhaps. But, if you're using a framework and/or good libraries, all of that underlying work and ecosystem is standardized for you. Now, the ecosystem with things like Symfony are really great.
Any benchmark that includes a JVM language always makes me look for a disclaimer that they allowed for proper JIT/warming time by running the benchmark a few times in the same VM instance before collecting the numbers. I may have missed it in this article, but I didn't see it.
Yes, startup time is a concern with the JVM, but that's not what it's built for - it's meant for long-runnning repeatable code paths.
There was nothing in the original design that said "only long running server processes". In fact, Java was originally aimed at set-top boxes, then web applets, then cross platform GUI apps. It was only after it failed in all of these that the server was discovered, despite the fact that "write once, run anywhere" is almost completely pointless when you control the hardware it runs on.
Any comments that "startup time is a concern with the JVM" always make me look for evidence that the person has made some attempt to investigate startup time for the specific circumstances they discuss.
A program written in C will execute faster if you run it multiple times, and slower if it has not been run recently. Presumably all of the benchmarks consist of many thousands of repetitions of the same process.
I see that the point of this site is supposed to be that a change in implementation (keeping the language itself held constant) greater affect on performance than rewriting in another language.
My problem with this site it makes poor support of that claim. The data is hidden behind loads of links, and none of the data is visualized at all. Even a few simple bar charts, keeping the hierarchy of the site otherwise unchanged would vastly aid comprehension. The points being made would come across much more easily.
This sort of data is the poster child for visualization, yet this site relegates the data to tables of numbers.
> The Go vs JavaScript lineup is quite stunning. Go destroys Node.js in memory usage and performance on every test except regex-redux (edge case?).
Why would it be stunning that a programming language compiled to assembler can beat a broken-by-design interpreted language which doesn't even have integers or arrays? The stunning part is that the people behind node.js have managed to get that much performance.
That's not a very fair or accurate description. JavaScript implementations use a JIT which is different than what people usually mean by interpreted. Types can be inferred without explicit type declarations. JavaScript has Array and integer semantics can also be inferred.
Ummmm, that's a somewhat harsh and arrogant reply. I actually enjoy coding in Node.js myself. I just wasn't aware how much faster go is (according to these benchmarks of course).
The JS language as specified is interpreted and doesn't have ints or first-class arrays; the code run by the VM that runs Node.js code is compiled and runs on smallints and arrays and so on.
(Not sure if you already knew this, but if so, you're not being factual, you're being selectively pedantic to make something sound worse than it is. Lack of ints or arrays isn't a big factor in JS performance for non-pathological cases.)
There's plenty to not like about JS, but being interpreted and not having integers or arrays is not high on its negatives list. The little bit of good far outweighs the whole lot of bad.
If you don't enjoy JS, it's because you haven't taken the time to learn what's great about it and have gotten hung up on the stupid things other people do with it or you're copying their mistakes.
> If you don't enjoy JS, it's because you haven't taken the time to learn what's great about it and have gotten hung up on the stupid things other people do with it or you're copying their mistakes.
oh come on now, it's perfectly ok for people to dislike a language for any number of reasons. I don't hate JS but I don't particularly enjoy it, and that could be said for a number of popular languages. it has nothing to do with "taking the time" to learn anything - some people just don't jive with JS and that's totally cool.
That last paragraph is a logical fallacy, because it posits that one's distaste for JavaScript could only come via ignorance. I will offer my own anecdotal experience to refute that assertion. I've been forced to use JavaScript since the 1990s, and have done significant amounts of work in Node for several companies. I dislike Node and JavaScript because JavaScript has a weak standard library, ugly syntax, is slow, is a pain to scale, and a pain to write complex code with. I will absolutely give credit to the Node community for improving JavaScript, in both speed and syntax, and admit that JavaScript is better now than it ever has been; however, so are most popular languages. Personally, I've removed JavaScript from my resume, and turn down jobs that use it. If I have to use it for some reason, I can, but right now, I typically avoid it altogether by using Elm, and real server-side languages on the backend. I can't speak for anyone else, but I know JavaScript and still don't like it.
I don't know, I rank not having integers pretty high on the list of faults. Wasn't there just a huge debacle with node being unable to represent inodes properly?
I think you're in the right here, but that's a little too far. I don't enjoy programming in JS, because I really like strong typing. And that's ok. You seem to be implying that everyone who doesn't like JS is deluded or arrogant, but that's not the case.
> There's plenty to not like about JS, but being interpreted and not having integers or arrays is not high on its negatives list.
We are having a discussion is about programming language benchmarks.
A benchmark in this case is a measure of how fast a computer program can run. A computer program is a series of instructions for a computer. These instructions are written in something called a programming language. A programming language which is interpreted is inherently slower than a programming language which is compiled to assembly language. A programming language which does not have integers but must represent all numbers as floating point numbers is inherently slower than a programming language which has integers. A programming language which does not have arrays, but must represent arrays using hash tables, is inherently slower than a language which does have arrays. Thus, Node.js, which is interpreted and does not have arrays or integers, would be expected to be have much lower benchmarks than Go's, and yet node.js has exceptionally high performance.
It looks like the JS version is using the built in runtime's regex engine.
I don't know exactly why (maybe differences or missing features in go's regex) but the go version is actually using the c FFI to call pcre.h, and without profiling it I'd guess a huge hit is that alone.
I think Go's regex lib is native Go, but simply naive. In any case, this particular benchmark isn't useful unless your application is regex as a service.
"If you're interested in something not shown on the benchmarks game website then please take the program source code and the measurement scripts and publish your own measurements."
Binary-trees is allocation bottlenecked, and Java has bump pointer allocation where Go has slow allocation. The root cause being that Go's GC is non-moving so it can't do bump pointer allocation without fragmentation, so it has to do a whole bunch of computation to find a place for each node.
Wow, I expected Erlang to score better, considering the claims of Elixir community how fast it is... compared to Ruby. OTOH maybe that tells more about Ruby.
Erlang makes no claims about being fast in these sorts of tests. They aren't using what it was designed to do.
Also some languages like, Ruby, do really well on some of these tests because the C-code underneath has been written well (example regexes are just as fast under ruby as they are anywhere else because the underlying C-library kicks ass though I forget the name of it).
Erlang's claim to fame at least for me is parallelism. When you scale out to thousands upon thousands of clients. Think about the simple fact it was using all cores on a processor out of the box by design that a computer had available. There's also the whole zero downtime aspect. In Erlang you should be able to design software (again downtime from the software side, not much you can do about hardware failure or server maintenance updates to your OS) that if it needs an update it can update itself without killing off anyone connected to it. There's also it's functional aspects.
This is why benchmarks really don't mean much. They are a myopic view of the complex nature of language selection. For example, I would never recommend Java for a bulletproof app that had to scale, because OTP is so much better at scalability, fault tolerance, live debugging, and hot code deployments. At the same time, I would never opt for Elixir for doing screen-scraping, because it is slow at it. Sometimes, developer productivity is more important than speed and scalability. It is important to know the strengths and weaknesses of various languages, and the most important aspects of your project, in order to pick the right tool for the job, I think. Performance benchmarks are but a small part of the selection process.
Good luck on seeing D on that site. There seems to be a active war between the sites maintainer and D. There used to be D benchmarks on the old site and people tries to release new code to the site, but for some reason the current maintainer has a issue with D.
And he thinks its no unfair that D is not resented when a lot of people look at his site and think: "If your not here as a language, then the language must be bad".
Good luck on changing his mindset because i have seen topics going back years regarding D not being there.
> one should be able to make any real-world task take about the same time in any two languages.
I don't think that's a practical claim. Interpreted or JIT compiled languages depend on the interpreter or VM. JavaScript has become much faster over time as JavaScript VMs have improved, independent of the details of any particular JavaScript program. Ruby has become faster over time for the same reason. I've seen Ruby programs I've written speed up significantly without any significant code changes when I moved from an older to a newer version of Ruby.
