[gary4gar]

Sad fact is MRI even with no matter how much effort we put in would never would be able to match GCs of battle tested VM like JVM.

so I am wondering, it makes sense to stop designing a VM & just focus on language. Perhaps, something like Truffle can be used to do the optimizations for you. Ruby on truffle is already 4-5x faster than MRI. And the ruby implementation was done by an intern in just 6months. All because power of JVM.

        https://twitter.com/headius/status/362616159897534465
        http://www.oracle.com/technetwork/java/jvmls2013wimmer-2014084.pdf

Something to keep in mind.

[vidarh]

The numbers for Ruby on Truffle are meaningless until people have had a chance to hit it hard with all the particular oddities of Ruby. Consider their 6 months was to hit 45% of RubySpec. You can reach 45% of RubySpec fairly easily if you go for the softest targets (I'm not saying that's what they've done - I haven't checked).

[EDIT: I see they're doing some interesting things that certainly ought to beat MRI. If I understand it correctly it seems like they are somehow collapsing type checks for multiple operations. Of course the devil is in the details - if they are trying to defer type or method checks, and throw away results if the checks fails (which should be rare) that will only be safe if modifications that does happen does not introduce or remove side effects that can't be "rolled back", but I might be misunderstanding their presentation]

The problem is the multitude of bizarre things that are legal Ruby. Like people doing eval("class Fixnum; def + other; 42; end; end;"). Yes, that's legal, and yes that means any integer arithmetic in your application is suddenly broken. More importantly it means any optimisations based on your beliefs about what any piece of code is meant to do, while they are most likely right, can turn out to be horribly wrong and so are problematic for a VM or compiler, without substantial amount of logic to be able to detect or bail out from optimised code to safe fallbacks. Doing so without slowing down the code when your guesses are right is hard because of how many ways there are of changing the behaviour of code in Ruby.

Unless your compiler understands eval() and it is possible for it to reason about the contents of the eval string, it can make pretty much zero guarantees about the state of the world after an eval() call, and so it can make pretty much zero guarantees about the state of the world after any method call that could reach such an eval() call.

Admittedly, that's a stupid thing to do, but it's legal in Ruby, and while the above example is extreme, you do find a lot of use that is roughly equivalent. E.g. autoload creates as much lack of predictability as eval. So does a 'require' or 'load' that might get triggered later in execution, for example.

The reason those are important is that it makes a massive amount of optimisations far harder: You can't blindly cache method pointers, for example, because any method call potentially invalidates them. You can't even cache class pointers, because they can change: You can return from a method call and suddenly an object has an eigenclass. You can't inline functions without guarding them somehow to fall back to the full method call when it turns out some idiot did redefine Fixnum#+. You can't assume seemingly "safe" stuff like Fixnum#+(some other Fixnum) will even return an object of the type you assume, for the same reason - someone might decide to implement a DSL that redefines it.

Frankly, it'd be fantastic to start deprecating some of the more obnoxious things like these, and weeding out the few uses of them, but as it stands today, a fast Ruby subset is "easy". A fast complete Ruby implementation is an entirely different beast. A fast incomplete Ruby implementation that refuses to support some of the most noxious corner cases would still be extremely useful for a lot of people, though.

(in the interest of disclosure since I'm talking about another Ruby implementation: I'm writing a series on my own slow process of writing a Ruby compiler, though my goals are very different - mostly focused on writing about the process)

[chrisseaton]

I'm the author of Ruby on Truffle.

I'll talk you through exactly how we solve the problem of redefining Fixnum, as one example of how we've tackled these problems.

Whenever you use Fixnum#+ in one of your methods, we lookup what that method is and cache the method so we can call it quickly next time. We actually never again check that this cache is still valid. The trick is that we sort of do the opposite - any time you do something that could invalidate that cache, we find the installed machine code that uses it, and delete it. If the machine code is still running somewhere on some stack for some thread or fibre, we jump from the machine code into an interpreted version which looks up the method again and carries on.

So Kernel#eval makes no difference - if something that you eval ruins your later cached method calls in the same method, that's not a problem because if you're still running the same machine code, then you can't have redefined Fixnum#+. If you had redefined it, you'd be back in the interpreter getting ready to compile again with new caches.

I'll also just point out that running RubySpec means we are successfully running something like 5000 lines of off-the-shelf unmodified systems code, just for the harness before we even get to the tests.

Our theory is that we can make Ruby very fast, without having to forgo any of your favourite random dynamic monkey-patching features.

Watch the video: http://medianetwork.oracle.com/video/player/2623645003001

Join us on the mailing list: http://mail.openjdk.java.net/mailman/listinfo/graal-dev

[VeejayRampay]

Would you mind providing some "grand order of things" hand-wavey estimate as to when exactly the public can expect to have Fast Ruby ™ © ®?

Also, will it the very same Ruby we all know, compatible with everything? i.e. will it be the Christmas I envision?

[chrisseaton]

I'm afraid I can't - sorry. Keep an eye on the mailing list or follow me on twitter (@ChrisGSeaton) though.

[VeejayRampay]

Done, thanks. And good luck to you guys.

[coldtea]

>Unless your compiler understands eval() and it is possible for it to reason about the contents of the eval string, it can make pretty much zero guarantees about the state of the world after an eval() call, and so it can make pretty much zero guarantees about the state of the world after any method call that could reach such an eval() call.

That's also true for Javascript, but it hasn't stopped three separate teams (Mozilla, Apple, Google) making it crazy faster than Ruby.

[vidarh]

First of all "Ruby" is not an implementation. The performance differences between MRI and the different patched versions of it, MacRuby, MagLev, jRuby, Rubinius etc. can be substantial.

[edit: I referenced https://github.com/cogitator/ruby-implementations/wiki/List-... here, but then I read through the rest of the list, and too much of it is "junk", so I added the list of the more mature implementations above instead]

But the point is not that making speed improvements is impossible. I strongly believe Ruby can be made as fast as current JS implementation or better.

But there's a vast gap between being able to, and for performance numbers based on 45% of RubySpec to mean anything about what we can expect to see in terms of performance of this particular implementation.

I'm working on a Ruby compiler myself (woefully incomplete; certainly vastly faster than MRI on the tiny subset it can compile, but pointless to benchmark for exactly the reasons I stated: I have no way of telling how much performance it'll lose to deal with method invalidation etc.), and I absolutely love that more people try to implement Ruby and it'd be fantastic if these performance gains stay as they flesh it out.

As someone else pointed out: Several Ruby implementation started out with impressive numbers on some subset of the language. Then they slowed down more and more as code was added to handle the corner cases of the language.

Maybe these guys will do better. Maybe they won't. The current benchmark does not tell us either way.

[coldtea]

>First of all "Ruby" is not an implementation.

Sure, but I was obviously reffering to MRI. Not to mention that, back in my day, Ruby WAS an implementation.

>But there's a vast gap between being able to, and for performance numbers based on 45% of RubySpec to mean anything about what we can expect to see in terms of performance of this particular implementation.

Sure I agree.

>As someone else pointed out: Several Ruby implementation started out with impressive numbers on some subset of the language. Then they slowed down more and more as code was added to handle the corner cases of the language.

I wonder then, if all that talent is not better served by implementing a BETTER version of the core 50% or 70% Ruby language as a new language, getting rid of edge cases and garbage. Perhaps add some good stuff from Python in for good measure.

Matter of fact, didn't Matz do something like that, with an embedded-oriented Ruby like language recently?

[dragonwriter]

> I wonder then, if all that talent is not better served by implementing a BETTER version of the core 50% or 70% Ruby language as a new language, getting rid of edge cases and garbage.

No, if you want languages that are less expressive but faster, there are plenty of options available. The thing is, MRI itself keeps getting faster implementing the whole of Ruby. The fact that new Ruby implementations that implement the easiest bits first tend to start out much faster than the mainline interpreter on code that only uses those most-straightforward-to-implement bits but then tends to converge closer to the speed of the mainline interpreter as it the implementation gets more complete doesn't mean that it would be better to make a new language. Much of the experience of those alternative interpreters is relevant to making improvements in the mainline interpreter that speed up, or otherwise improve, "complete" Ruby (including times when the alternative interpreter becomes the mainline interpreter, as occurred with YARV, which was an alternative interpreter before it became the mainline interpreter in 1.9.)

> Matter of fact, didn't Matz do something like that, with an embedded-oriented Ruby like language recently?

If you are referring to mruby, that's an embedded-oriented Ruby implementation, not a Ruby-like language.

[vidarh]

Losing too much of Ruby would make it pointless. But there are certainly edge cases that we could lose readily and not care very much.

When was the last time you saw anyone redefine operators on Fixnum for a good reason, for example?

My pet peeve is small things like freezing many of the base classes by default, for example. As well as a "meta programming module" in the standard library that'd gather up as much as possible of what people (ab)use eval() for today in specific, narrowly tailored methods that implementations could provide specialised versions of. A limited "general purpose" eval() that is not allowed to modify the class hierarchy would be another good thing - anything that'd combine to allow implementations to defer type and method cache guards and invalidations as long as possible would make it far easier to improve performance, with very little impact on most developers.

[coldtea]

>No, if you want languages that are less expressive but faster, there are plenty of options available.

I'm not sure there are in the style we're talking about. Only Lua comes to mind. Maybe I'd add Julia there too.

A modern Python/Ruby replacement, built for speed and with a large-ish community would be nice to see. Even with static inferred typing.

It's not like we can't have new languages anymore. After all both Ruby/Python came out of nowhere around 1992-4, a time where there was no modern web and even less resources to grow a language.

[VeejayRampay]

The Ruby developer in me wants to have his lunch and eat it.

I'm sure there's a way to have the crazy whole that is Ruby AND the speed. Dammit, Javascript is fast now and it's not like it was more optim-ready than Ruby is to begin with.

[vidarh]

> I wonder then, if all that talent is not better served by implementing a BETTER version of the core 50% or 70% Ruby language as a new language, getting rid of edge cases and garbage.

I would very much like to see that. But I think we first need more competition to MRI.

E.g. note the proposal for "refinements" for Ruby 2.1 which is an implementation nightmare for anyone that wants a fast implementation. See Charles Nutter's (of jRuby fame) comments about it, for example. Since MRI is as slow as it is, there's little incentive to keep features like that out - it won't kill MRI performance.

It's also hard to determine exactly what the viable subset should be, since most those of us who would like a faster Ruby also love Ruby a lot because of how dynamic it is, when it is used right, and jump ship if the language lost too much flexibility in the quest for that performance.

Part of my own motivation for writing about writing a Ruby compiler is exploring what parts of Ruby can be implemented efficiently because frankly it's hard to even guess exactly how an "efficient Ruby" should look.

There are some obvious problems for implementations that want to boost performance, like too much reliance on eval() and defining eigenclasses on objects, as well as autoload, require and include, all of which can worst case trash all method caching and optimisations all over the place.

But throwing out all of that would be brutal, especially given common Ruby patterns likes dynamically require'ing everything in a directory at application startup, which should be fine, vs. a "require" occurring later in execution. And it's not clear that there aren't other common patterns that'll cause a lot of pain.

I think we will see at least implementations with options to disable support for certain things, or with support to let applications declare "from now on, no shenanigans" to let the implementation take shortcuts, would be very helpful.

There's a lot of things developers could do themselves, that would let even a compliant implementation speed up. E.g. call #freeze on all classes you have no intention of modifying somewhere that is easily identified by relatively superficial analysis would make a massive difference (suddenly you can cache lots of extra methods, and even inline and unroll things like "each" loops in many cases that'd otherwise be an expensive flurry of method calls).

Other things are about developer practice: Freeze all objects you don't want to modify ASAP on creation. A good implementation could make good use of that too. But today there is no incentive for Ruby users to write Ruby that is amenable to fast execution because the implementations don't take advantage of it.

Once there is an implementation that makes the advantages of writing Ruby to a subset that is more amenable to fast execution, then I think it'd be possible to get traction for deprecating and removing features that are a performance nightmare.

> Matter of fact, didn't Matz do something like that, with an embedded-oriented Ruby like language recently?

Sort of, though mruby appears to focus on size and ability to embed rather than specifically picking a subset that's amenable to a fast implementation. It's still a bytecode interpreter, for example, and it's not even aiming for complying with the (already limited) ISO standard for Ruby.

[Argorak]

Most Ruby implementations started with awesome numbers in a ridiculous short time. Remember the time when MacRuby was 6 times as fast as MRI? Those times are over after they implemented Ruby proper and ran real world programs.

[coldtea]

Haven't seem them getting slower in later stages, and surely not THAT slower. Any citations?

Their main problem is that the project got mostly abandoned in favor of the paid-for RubyMotion (which uses AOT compilation).

That said, Lua and JS are as dynamic and Ruby and both are plenty faster.

And PyPy is plenty faster than Python, despite being 100% compatible now.

[Argorak]

Yes, but PyPy was not written in 6 months. For a rather comprehensive (but not so recent) benchmark, see:

http://etehtsea.me/the-great-ruby-shootout http://igor-alexandrov.github.io/blog/2012/11/05/yet-another...

MacRuby looses against MRI in many benchmarks and MRI doesn't fare as bad as many people say.

[kingkilr]

PyPy wasn't, but as a result of the groundwork we laid in building it, I (and several contributers) were able to build Topaz in under a year: http://topazruby.com/

[Argorak]

How much work do you expect it to be capable to replace MRI?

[gary4gar]

Curious to know what are thing in "ruby proper" that make it slow?

[vidarh]

The short answer is that Ruby is "too dynamic" and "too malleable": An object can get a new class when you call a method on it, for example, or every method might be redefined, including seemingly safe ones like arithmetic on constant integers.

The longer explanation of this one issue (there are others):

Somewhere deep in the bowels of your code, some library calls eval(foo). "foo" is what exactly? If you can't track it back to a constant string, you're SOL from an optimisation standpoint: Any and all method pointers and class pointers you may have cached to speed up method calls are now unsafe, as for what you know, the next time you try to do "2 + 2" you might get "5". Or a string. Or your harddrive might get formatted - that latter is particularly important: Methods may change from not having side effects to having side effects or back, potentially in the middle of evaluating an expression.

99% of the time, the changes will not affect core classes, but even so a language implementation that wants to be complete will need to at least guard against it even for basic things like adding two numbers.

That means there's lots of stuff you can't cache, or where you need logic to invalidate caches that can be very complex, or you need to be able to validate any cached information. And you either need to do this for every method call, or you need to find ways of safely rolling back calculations (and you'd need to ensure you're not inadvertently triggering side effects that you can't roll back if you choose that option), or you need to be able to deduce what code can trigger these things, and tread with appropriate caution afterwards.

[Argorak]

For example method cache invalidation is a hot topic in dynamic languages that is often ignored in synthetic benchmarks. A Ruby example for this is:

  output.extend(ContentTyped)

(thats from Sinatra: https://github.com/sinatra/sinatra/blob/154859f1553b8bacea97...)

I am not saying that these things cannot be implemented fast, but are usually the things that get ignored in a first implementation and take a lot of work to get right and quick.

[vidarh]

Now I want to slap a Sinatra developer.

Note that the problem is not to implement the example you gave fast. That example almost certainly won't be: Implementing it without instantiating a new eigenclass just for that object and populating it with the right methods would be tricky, and that is not going to be particularly fast.

The problem is as you mention that method caches must be invalidated, affecting other calls that we otherwise would expect to be fast, and requiring overhead everywhere to prevent doing the wrong thing in the face of having the rug pulled from under you.

In this case it's fairly benign: ContentTyped "just" installs attribute accessors, and "only" affects that object, and sufficient bookkeeping could restrict the cache invalidation accordingly.

(Thanks, btw., it's an interestingly rare example of a real use of extending objects directly)

[Argorak]

The problem here is not just that it trashes the method cache for this object. In most versions of MRI, this trashes _all_ method caches, not just the ones for String.

A sufficiently clever JIT could find out that there is only ever a String instance extended (so, basically, the extension is monomorph) and introduce the resulting type.

However, this example is not rare. Some more:

The DCI pattern: http://www.sitepoint.com/dci-the-evolution-of-the-object-ori... (scroll to source code) ROAR, a popular representation gem: https://github.com/apotonick/roar

[vidarh]

It shouldn't need to cache method caches for String at all. It is not extending String. It is extending a specific instance of String (or whichever other class that object happens to be - for the rest of this I'm assuming it's a String), and thus not even touching the String class, but the eigenclass of the object:

  >> module ContentTyped
  >> attr_accessor :content_type
  >> end
  => nil
  >> foo = "bar"
  => "bar"
  >> foo.extend(ContentTyped)
  => "bar"
  >> foo.content_type
  => nil
  >> "another string".content_type
  NoMethodError: undefined method `content_type' for "another string":String
  	from (irb):31

Externally this eigenclass is not readily visible, but within MRI it is, and for the purposes of a method call, the immediate superclass of foo above is not String, but its own object-local class object.

If you are right this might be one of those areas where there's room for quick-wins. It doesn't take a "clever JIT" to localise this damage by ensuring the method cache is only invalidated for whichever entity is extended (which in this case is no existing class, unless the object has been previously extended).

As for your examples, I wonder if these people realize that their code is equivalent to instantiating a new class for every single object. To me ROAR for example just looks horribly conceptually broken. In effect they are creating a new class per object in order to add non-stateful behaviour, which just makes me want to rage

(EDIT: and DCI makes me want to rage too, for different reasons - they could achieve most of the same by composition by wrapping the objects in static classes instead of dynamically extending the objects; to me these examples stands as prime examples of how the horrible performance of MRI leads to people making implementation choices they'd never make with a Ruby implementation where the things that can be made fast are fast).

I don't care if that is never a fast case, as they have options that would be faster: include the representer modules in the resource class like Article, subclass and include, wrap it when needed. I want it to be easy to write fast Ruby code - I don't particularly care if pathologically crazy implementation choices remain slow...

[riffraff]

I wanted to find the reference but I failed.

Anyway, I just wanted to counter that IIRC mike pall of luajit fame has stated that in his opinion a VM/JIT laser focused on a single language's semantics is much easier and in the end more effective than using a general purpose one.

Alas, I can't find the reference, but the luajit example stands on his own. Whether designing your own VM is a better use of money&time is largely a trade off of what your targets are. E.g. HotSpot could have been a great target unless you are interested in fast startups, or whatever.

[coldtea]

>Sad fact is MRI even with no matter how much effort we put in would never would be able to match GCs of battle tested VM like JVM.

Nothing imposissible about it happening though. Just money and time.

[trailfox]

Ruby and Java were both released in 1995. It's now 18 years later and MRI can't keep up with a 1998 JDK. Could be a long wait...

[coldtea]

It's not about absolute time. I mean a GC worse than another 15 year older GC, does not necesarilly needs 15 years to catch up to it.

It might never catch up, or it might catch up in just a few years years, if people and funds are concetrated on that goal.

For example from the legacy of the usual dog slow JS implementations, we got V8 feature complete in less than 2 years and could run rings around them. Same for the JIT-ed js engines of Mozilla and Apple.

And while Google and Apple have tons of funds, Mozilla scale development is not that far from the reach of a typical open source project with industry bakers.