[Pewpewarrows]

Completely unscientific, but if these outputs are any indication, this is going to be great news for Rubby users in the future...

  $ time ruby -e "puts 'hello world'"                                                                                                                           
  hello world

  real    0m0.184s
  user    0m0.079s
  sys     0m0.092s

  $ time ~/Downloads/topaz/bin/topaz -e "puts 'hello world'"                                                                                                    
  hello world

  real    0m0.007s
  user    0m0.002s
  sys     0m0.004s

[raphael_kimmig]

There is a neural net example benchmark in the topaz git repo. Don't know how representative that example is, but at least startup time shouldn't be dominating the results...

  $ ruby -v
  ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-darwin11.3.0]
  $ ruby bench_neural_net.rb
  ruby bench_neural_net.rb  17,74s user 0,02s system 99% cpu 17,771 total

  $ bin/topaz bench_neural_net.rb
  bin/topaz bench_neural_net.rb  3,43s user 0,03s system 99% cpu 3,466 total

[ErikRogneby]

should be able to run any number of ruby examples from here: http://benchmarksgame.alioth.debian.org/u32/ruby.php

[pserwylo]

You should check out a tool I heard about recently called "ministat". An example of how to use it to benchmark multiple runs, and then compute the statistics from each run are available here:

http://anholt.net/compare-perf/

Example output:

  +------------------------------------------------------------------------------+
  |                   +                            x                             |
  |                   +                            x                             |
  |           +       +                            x x            x              |
  | +    ++   +++     +                            x xxx xx       x              |
  |++ ++++++++++++++++++                   x  x    x xxx xxx  xxxxx              |
  |++ ++++++++++++++++++ +++ +       ++   xxxxxxxxxx xxxxxxxxxxxxxxxx  xx x    xx|
  |     |______MA______|                        |________A________|              |
  +------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x  57      45.62364     46.437353      45.93506     45.951554    0.19060973
  +  57     44.785579     45.534727     45.042576     45.056702    0.16634531
  Difference at 95.0% confidence
    -0.894852 +/- 0.0656777
    -1.94738% +/- 0.142928%
    (Student's t, pooled s = 0.178889)

[grn]

In order to eliminate the potential bias of startup time I run similar test in many iterations. Here's what I got:

    $ time ruby -e "10000.times { puts 'hello world' }" > /dev/null
    real    0m0.102s
    user    0m0.096s
    sys     0m0.005s

and

    $ time ./topaz -e "10000.times { puts 'hello world' }" > /dev/null
    real    0m0.098s
    user    0m0.071s
    sys     0m0.026s

Any idea why I don't see such big difference?

[fijal]

topaz probably has a pretty slow IO (for bad reasons, it's an RPython problem)

[duaneb]

Because the majority of the cycles ran are probably in 'puts', which is implemented in C if I were to guess.

[emillon]

The overhead of starting the interpreter must be longer that executing the loop.

[rbehrends]

Because a puts loop is I/O bound, not CPU bound.

[fernandezpablo]

what is your 'ruby'?

[onedognight]

Good question. How many shells is rbenv/rvm executing? Do you get similar results with an absolute path?

[grn]

ruby that comes with Debian Squeeze.

    $ ruby --version
    ruby 1.8.7 (2010-08-16 patchlevel 302) [i486-linux]

[draegtun]

To scratch a quick interesting thought that came into my head I just had a look at Cardinal, which is a Ruby implementation running on the Parrot VM - https://github.com/parrot/cardinal

So downloaded & built Cardinal (which went seamlessly however I did have Parrot already installed) then I did same benchmarks alongside ruby1.8 here:

  $ time ruby -e "puts 'hello world'"
  hello world
  
  real	0m0.130s
  user	0m0.049s
  sys	0m0.071s

  $ time parrot-cardinal -e "puts 'hello world'"
  hello world

  real	0m0.057s
  user	0m0.037s
  sys	0m0.019s

Very interesting because I thought Cardinal was supposed to be slow!

I think more diverse benchmarks are required. And when time permitting I might add Topaz & ruby1.9 into the mix.

[Cowen]

Whenever I see frontpages for these kinds of projects like "a faster X" or "X written in Blub", the first thing I want to see on the frontpage is how this new project compares to X in terms of quality and performance. Even specious benchmarks would help more than zero benchmarks.

I wish more frontpages for these kinds of projects would do that.

[pyre]

If they put that on their frontpage, there would be at least 20 posts on here bashing them for it because they didn't get it right (or just accusing them of outright lying/incompetence).

[callum85]

"Even specious benchmarks would help more than zero benchmarks."

I disagree. Zero benchmarks is definitely better than specious benchmarks.

[Cowen]

To clarify, I was trying to use "specious" as a synonym of "flawed." I thought this was the common usage, but apparently not.

As to your point, obviously no one should be making any decisions off of flawed benchmarks, but flawed benchmarks (not so far as outright lies, just flawed) at least give me an objective justification to investigate further.

Even some flawed benchmarks could help turn the initial tide of responses like "this is X written in Blub, it's bound to be better!" or "this is a faster X! Now everything will be twice as fast!" They're silly examples, but it seems like every time a new technology comes out, these are the kinds of knee-jerk, overly-optimistic reactions people tend to have.

[kenko]

" To clarify, I was trying to use "specious" as a synonym of "flawed." I thought this was the common usage, but apparently not."

"Specious" does mean, more or less, flawed.

Zero benchmarks are better than flawed benchmarks.

[nessus42]

All benchmarks are flawed in one way or another.

[amalag]

spe·cious /ˈspēSHəs/ Adjective Superficially plausible, but actually wrong: "a specious argument". Misleading in appearance, esp. misleadingly attractive: "a specious appearance of novelty".

So you are saying you would prefer wrong information?

[PuercoPop]

http://www.merriam-webster.com/dictionary/specious

1 obsolete : showy 2: having deceptive attraction or allure 3: having a false look of truth or genuineness : sophistic <specious reasoning>

So instead of taking the common meaning of specious, a deceptively attractive benchmark we have to go to a less common use of specious in order to construct an specious argument about the proper use of specious? Nevermind that it is distracting from the main point of the discussion about benchmarks in the context of Topaz.

[losvedir]

For what it's worth, amalog's definition of "specious" is the one I'm familiar with. My girlfriend recently quizzed me on her GRE words with that one, and my definition was the given one, too.

So I'd argue that amalog's definition IS the common one.

[tomcorrigan]

It depends where you look for the common meaning. To native English speakers in the UK, Australia, New Zealand (ie outside North America, coincidentally including the country where the English language developed) specious has one very clear meaning

I would also direct you to the usage examples in your link. All of which use specious in a context that implies deception of outright falsity. I have never ever seen specious used synonymously with obsolescence.

[svachalek]

I read that as "showy (obsolete)". Either way, amalag's says "wrong", the other says "deceptive" and "false", I'm not sure what anyone's arguing about. Specious data has no value.

[victorh]

I disagree with your argument! If inventing the language meant you got to choose all the definitions we wouldn't have English in the first place. Shift happens. I don't think this particular usage is common, though.

[Cowen]

I was using "specious" as a synonym of "flawed." Perhaps that was the incorrect usage.

But in the case of advertising a new library/project, which is arguably one of the main functions of the frontpage, flawed benchmarks (though not so far as outright lies) at least give me an objective reason to investigate further.

With no benchmarks, generally I'll open the page, mutter "that's nice" and move on with my business. I'd imagine I'm far from the only person who does that. Young projects don't help themselves when they don't effectively advertise themselves.

[haven]

Expanding on the concept of entirely unscientific benchmarks, I benched some simple prime number math with Topaz, Ruby, JRuby and RBX.

Looks promising for Topaz! https://gist.github.com/havenwood/4724778

[seivan]

Try comparing with Rubinius

[eranation]

Impressive, how does it compare to JRuby?

[andrewaylett]

That's probably not a fair test to use on JRuby, as the JVM is notoriously slow to start.

[JoshTriplett]

That doesn't make it unfair to JRuby, it just means JRuby will probably lose that benchmark. :)

[nirvdrum]

If the goal is to benchmark Ruby execution time, it's unfair. The slow startup time is a valid concern when considering short-lived sessions, but it's kind of meaningless when trying to benchmark a Ruby implementation.

[fleitz]

There are no 'fair' benchmarks, all benchmarks should be biased to the problem you're actually solving running your actual workload. If you can't replay your workload at multiples of real volume then you should probably work on doing that before benchmarking as it helps you out with the real problem of verifying your infrastructure.

In general a benchmark is probably the worst metric you could ever use for deciding on an implementation. Unless the profit margin of your business is razor thin and dependant eeking out every last drop of performance, and even then most of those gains will be from extremely small sections of code that are probably best written in assembler by a programming God, and you should investigate FPGAs, ASICs, and other high performance solutions.

If your benchmark (infrastructure) involves a database (or anything that uses disks) that's probably going to be the problem long before the speed of your language / language implementation.

[nirvdrum]

I don't even know where to begin with this.

In all comparisons, you should remove confounding variables. Yes, you should benchmark something you actually care about, otherwise what's the point? That doesn't mean all other variables are immediately null and void. That's why I said said if your goal is to measure ruby execution time, you should remove startup time.

As for the practice of benchmarking in general, you're partially right. Micro-benchmarks are usually useless because they don't map to real work load. But profiling and speeding up small portions that are used heavily can have drastic improvements that in isolation seem small -- the death by a thousand cuts problem. Not all improvements come from isolated instances with very slow performance profiles.

This fallacy about DB access and not needing to optimize really needs to go away though. Even if 50% of your app is spent hitting DB, you have opportunity to speed up the other 50% and it's likely far easier. Ruby in particular is ripe for improvements on the CPU side. I managed to reduce my entire test suite time by 30% by speeding up psych. I managed to cut the number of servers I need in EC2 in half by switching from MRI, Pasenger, and resque to JRuby, TorqueBox, and Sidekiq. And I've managed to speed up my page rendering time anywhere from 8 - 40x by switching from haml to slim. None of these changes required modifications to my DB, none required me to write assembly, none required me to switch to custom-built hardware, and each helped reduce the expenses for my bootstrapped startup, while improving the overall experience for my customers.

[__david__]

It's not meaningless if you're trying to make a command line app and not a Rails app. Startup time does matter for command line apps (a lot!).

[nirvdrum]

Agreed. But then you're not benchmarking Ruby execution speed, which really makes comparing the two not very worthwhile. By the same notion, then when you compare on something like JRuby, you really should be comparing JDK 6, 7, and 8 builds, along with various startup flags, both JVM and JRuby. E.g., short runs may benefit greatly from turning off JRuby JIT, JAR verification, and using tiered compilation modes. May as well add Nailgun and drip in there as well, since both can speed up startup time on subsequent runs. Which one of these is going to be the JRuby you use in your comparisons? Or are you really going to show 20 different configurations?

You can run into the same problem with MRI and its GC settings. If too low for your test, you're going to hit GC hard. It's best to normalize that out so you have an even comparison. Confounding variables and all that.

There's a lot you can do outside the Ruby container to influence startup time. When comparing two implementations, the defaults are certainly something to consider, but not when trying to see which actually executes Ruby faster. They are two different metrics of performance and should be compared in isolation.

[freehunter]

Well startup times of the JVM aren't terribly relevant if, for example, your app doesn't need to start the JVM every time it's used.

[SkyMarshal]

In production server apps the JVM always running anyway, and under some degree of Hotspot optimization, so for a JRuby benchmark to be informative and worth anything to you, you'll want to account for that.

[charliesome]

How the hell is your Ruby that slow to start?

    $ time ruby -e "puts 'hello world'"
    hello world
    
    real    0m0.011s
    user    0m0.008s
    sys     0m0.003s

[d33pika]

The first time I ran:

   time ruby -e "puts 'hello world'"
   hello world

   real	0m0.221s
   user	0m0.005s
   sys	0m0.006s

subsequent times:

   time ruby -e "puts 'hello world'"
   hello world

   real	0m0.008s
   user	0m0.005s
   sys	0m0.003s
 

So, I guess he ran ruby first followed by topaz and ended up with those results

[snissn]

    $ time ruby -e "puts 'hello world'"
    The program 'ruby' can be found in the following packages:
     * ruby1.8
     * ruby1.9.1
    Ask your administrator to install one of them

    real	0m0.060s
    user	0m0.040s
    sys	0m0.016s

[axomhacker]

Don't know if you are trolling. But if this is genuine: this is rbenv telling you that you have multiple rubies installed.

1. Pick one (just for this session): $ rbenv shell ruby1.9.1

2. And then run the example.

By the way 1.9.1 is really old already, 1.9.3 has a lot more bug fixes.

[alrs]

ruby1.9.1 on Debian isn't ruby 1.9.1.

The Ruby language changed between 1.9 and 1.9.1, so a new package name had to be created.

If 1.9.1 was just called "1.9" it would break all of the packages in Debian that depend on whatever language features were different between 1.9 and 1.9.1.

"ruby1.9.1" in Debian 7.0 provides version 1.9.3.194.

[axomhacker]

The OP's output looked more like rbenv's output though, right? How did you figure that to be debian's message?