[chromatic]

Parrot was always intended to support multiple languages, from Simon's original April Fools joke to Larry's dual desire to run both P6 and Perl in the same process on the same VM as well as to make CPAN available to all dynamic languages in the same process on the same VM.

That's why Dan and Guido had a bet that Parrot would run Python faster than CPython, and that's why Parrot attracted contributors like Sam Ruby.

It's definitely true that, as P6 dragged on and burned out a lot of contributors (including multiple project managers), many Parrot contributors thought that coupling Parrot's success to P6 was a mistake, but I can't see that as a change in focus because the multi-language focus was a goal from the start.

Granted, Parrot did wander in the wilderness for a while waiting for the P6 design process to proceed, and Parrot did evolve in some unfortunate directions because of that. The rest of this comment covers 2007 onwards though.

Your question about components is particularly interesting, because of what really happened.

A couple of P6 developers wrote PCT, a set of tools and libraries to make developing compilers and languages on Parrot easier. This was a good thing, and Parrot users and developers started migrating their language implementations to that toolkit.

The PCT developers refactored and revised this a couple of times, and to make that easier, eventually they took the code out of the Parrot tree and started syncing PCT into the Parrot tree on a semi-regular basis.

I was working with them weekly to fix Parrot bugs, improve Parrot performance, and add Parrot features to reduce the amount of code necessary in PCT. Thus it's difficult for me to believe that the Parrot developers were designing and implementing Parrot away from P6.

Toward the end of 2010, the P6/PCT developers said they were going to rewrite PCT to support multiple VM backends. I argued this would create a lot of churn in the language implementations actively using PCT (ironic, because the PCT developers spent a lot of time before and after this complaining bitterly that they couldn't keep up with Parrot churn) and would provide very little value for Parrot (PCT was out of tree already, and syncing in code designed explicitly to undercut Parrot seemed like a stupid idea). These discussions all took place in the public Parrot IRC planning sessions, which were logged.

I also had concerns about yet another rewrite undercutting P6, especially coming six months after the underwhelming Rakudo Star release, but it was clear which direction the winds were blowing, and that's when I stopped working on both P6 and Parrot.

Five years later, P6 had its first official release and Parrot was well buried.

I know this contradicts the official Raku hagiography, but I've gone back and read the primary sources a few times over the years as people have asked about it, and I'm confident the record backs up what I've said.

[lizmat]

Well, you are definitely more informed about the pre-2012 period. I only really got involved from 2012 onward, so I have no first hand knowledge about that period, except maybe a few weeks in 2000 (when I was dragged in by Johan Vromans) and 2005 (when I was dragged in by Audrey).

I just know second-hand that something broke the trust between the Perl 6 and Parrot developers around the first Rakudo Star release. And that strengthened the notion that another backend for Perl 6 was needed.

> I know this contradicts the official Raku hagiography

OOC, which official hagiography are you referring to?

Also, I find the use of "hagiography" in relation to Raku a bit strange: if anything, with the rename, the last saint has basically left the building?

[raiph]

> I only really got involved from 2012 onward, so I have no first hand knowledge about that period, except maybe a few weeks

I'm in roughly the same boat. I too dipped in and out starting in 2000, and had been reading contemporary discussions on and off from around 2009.

I want to draw your attention to some things I know due to publicly available records, things I think shed light on what really happened.

The first thing is that Rakudo began technically shifting to a multiple backend architecture in mid 2009:

* Following Jesse Vincent's summer 2009 discussion with Patrick Michaud about Perl 6 running on JVM and .NET, NQP was rewritten in the fall of 2009 to support multiple backends.[0]

* jnthn edited his website to say his first interest in working on Perl 6 going forward was "Transforming Rakudo from a single backend compiler to one capable of targeting multiple backends".[1]

> I just know second-hand that something broke the trust between the Perl 6 and Parrot developers around the first Rakudo Star release. And that strengthened the notion that another backend for Perl 6 was needed.

As explained above, Patrick had already focused his and jnthn's efforts on a multiple backend Rakudo in mid 2009, a year before Rakudo Star.

And, as chromatic has said, and the record shows, commits to Parrot "fell off a cliff" in early 2011, about 6 months after Rakudo Star, and a year before the dates of the first commit timestamps in MoarVM's git history, in Q1 2012.

----

[0] https://www.youtube.com/watch?v=XgPh5Li3k4g#t=2m

[1] http://web.archive.org/web/20091218092004/http://jnthn.net/p...

[raiph]

> Weird how almost nothing Patrick talked about in that talk panned out -- no JVM backend ... no JavaScript backend

At the time of writing this comment:

js Use new nqp::time instead of nqp::time_(i|n) 20 days ago

jvm [JVM] Add op 'nativecallinvoke' 15 days ago

moar Disallow explicity specifying op write registers 8 days ago

From https://github.com/Raku/nqp/tree/master/src/vm

[chromatic]

The first thing is that Rakudo began technically shifting to a multiple backend architecture in mid 2009

Weird how almost nothing Patrick talked about in that talk panned out -- no JVM backend, no CLR backend (the one that Jonathan talked about in that blog post too), no JavaScript backend, no Tcl, no Perl, no Ruby, no Python frontends.

In fact, it's amazing how that fourth rewrite of NQP ended up not working at all for any of the backends Patrick showed on his slide. Or at least not working yet.

the record shows, commits to Parrot "fell off a cliff" in early 2011, about 6 months after Rakudo Star

How strange that that happened just after the developer summit where I said that the NQP strategy was going to leave Parrot developers in a dead end. Imagine that -- people don't want to stick around on a project that's continually undercut by bad technical decisions.

Thanks for finally confirming what I've been saying for years.

[lizmat]

> that the NQP strategy was going to leave Parrot developers in a dead end

If Parrot had provided all that Perl 6 needed at that time, I don't think the multiple backend strategy would have been selected. Why would one?

> where I said that the NQP strategy was going to leave Parrot developers in a dead end

To me that feels like a self-fulfilling prophecy. Also, had Parrot been able to attract other languages to a significant extent, the abandoning of Parrot by Perl 6 would not have been an issue.

Personally, I think Parrot was not able to attract other languages because:

1. it didn't provide enough benefits to account for the extra effort needed by language developers. It definitely did not for Perl 6 at the time.

2. it didn't have a core team welcoming enough to keep anybody vaguely interested in Parrot

In the end, it is always about being able to create a community in open source. Without it, a project is indeed on a dead end.

[chromatic]

If Parrot had provided all that Perl 6 needed at that time, I don't think the multiple backend strategy would have been selected. Why would one?

Given that the multiple backend strategy only succeeded in replacing an existing backend with another backend several years later without achieving any of the interoperability goals of the JVM or CLR, I'm not even sure how to speculate on the reasoning behind that decision.

Also, had Parrot been able to attract other languages to a significant extent, the abandoning of Parrot by Perl 6 would not have been an issue.

You haven't addressed my point that NQP yanking the rug out from most other languages running on Parrot harmed Parrot. That seems like a material concern.

it didn't have a core team welcoming enough to keep anybody vaguely interested in Parrot

You haven't addressed my point that I personally spent multiple months offering to make Parrot better fit NQP/PCT for P6 and was repeatedly rebuffed.

[b2gills]

> Given that the multiple backend strategy only succeeded in replacing an existing backend with another backend several years later without achieving any of the interoperability goals of the JVM or CLR, I'm not even sure how to speculate on the reasoning behind that decision.

Umm. It runs on MoarVM, JVM, and JavaScript. How is that not multiple backends?

If I remember correctly, the JVM port predates the existence of MoarVM.

Granted the JVM port has only ever been a second tier backend. Partially because it only supports Java's level of Unicode support.

---

> You haven't addressed my point that I personally spent multiple months offering to make Parrot better fit NQP/PCT for P6 and was repeatedly rebuffed.

Yeah, because that really wasn't the main problem.

The main problem with Parrot was that the intermediate NQP/PCT code supporting it needed a complete rewrite.

The intermediate code for supporting MoarVM, JVM, and JS are all very similar. They are also significantly different to how Parrot was handled.

There is relatively little backend specific code in Rakudo itself for the three current ones. There was a lot of backend specific code in Rakudo for Parrot. Instead the vagaries of the different backends is mainly handled by NQP.

That means that if you made Parrot work better the way Rakudo-on-NQP-on-Parrot was at the time, it would have been a wasted effort.

---

Imagine the effort you were talking about doing was like tunnelling through a mountain.

The problem is that Rakudo-on-NQP-on-Parrot was going around the mountain instead.

Sure you would have reduced the total length. The problem is that you would have to create a new tunnel in a different direction once Rakudo-on-NQP-on-Parrot instead started tunnelling straight through the mountain to meet you.

Pretty sure that's why they told you to not do that.

[lizmat]

Given that the multiple backend strategy only succeeded in replacing an existing backend with another backend several years later without achieving any of the interoperability goals of the JVM or CLR, I'm not even sure how to speculate on the reasoning behind that decision.

I'd say, it turned out to be more difficult than expected?

You haven't addressed my point that NQP yanking the rug out from most other languages running on Parrot harmed Parrot. That seems like a material concern.

I guess I don't understand how NQP would be yanking the rug out from other languages, technically speaking? That it could be seen as a vote of no-confidence: well such is life in the open source world.

You haven't addressed my point that I personally spent multiple months offering to make Parrot better fit NQP/PCT for P6 and was repeatedly rebuffed.

Not having been around at that time, I don't think it is up to me to address that point. If you were rebuffed, I can only surmise that interpersonal and technical relationships had already soured enough not to trust the validity of your offer. Note: I'm not saying that the intent of your offer was questioned.

[chromatic]

I just know second-hand that something broke the trust between the Perl 6 and Parrot developers around the first Rakudo Star release. And that strengthened the notion that another backend for Perl 6 was needed.

If I recall correctly, that's about the time Jonathan started complaining that the Parrot object model didn't work very well for P6 and wrote his own for PCT, which is still weird to me because he was the primary implementer of the Parrot object model in Parrot. (I don't remember the dates, but he also wrote an object model in C# and another in Java, which you mentioned in another comment.)

I tried for a long time to migrate the PCT object model back into Parrot, but both Jonathan and Patrick told me repeatedly it wasn't a priority, so I stopped offering.

I'm not the only person who had this experience; I remember someone else having a similar experience with Unicode support.

OOC, which official hagiography are you referring to?

The one that says "Parrot dumped P6, so the P6 developers had no choice but to write their own VM".

I was there. That's not true at all.

[microtherion]

As a further clarification of my earlier remarks, when I was talking about Parrot developers not being interested in Perl 6 at all, I was not referring to your era, but to considerably earlier times: https://www.sidhe.org/oldblog/archives/000435.html

Different eras with different dysfunctions, and different people involved, but for some reason, always personality clashes.

[chromatic]

That was my era too, but it's definitely true that Dan lost interest in P6 well before he stepped down, and Leo never seemed to have a specific interest in P6.

[lizmat]

> The one that says "Parrot dumped P6, so the P6 developers had no choice but to write their own VM".

I don't think that I ever implied that. To me, in one sentence, it's more like "Parrot and Perl 6 diverged, so that Parrot was no longer a good fit for Perl 6, so the Perl 6 developers had no choice but to write their own VM"

[chromatic]

Parrot and Perl 6 diverged

To me this reads like the "mistakes were made" phrasing used when someone wants to be technically accurate but not say anything meaningful.

Regardless of whatever Raku developers and advocates say, I, as a lead developer of Parrot, went out of my way to make it work better for P6. Patrick asked me not to do certain things and so I didn't do those things.

Maybe that was my mistake.

But it's categorically untrue that Parrot wanted to get rid of P6 or considered P6 an afterthought or wasn't willing and able to be receptive to P6's needs, and I'm disappointed that that untruth continues to persist.

[lizmat]

Regardless of whatever Raku developers and advocates say, I, as a lead developer of Parrot, went out of my way to make it work better for P6.

May I point out that basically nobody of the then Perl 6 team (except Jonathan) is currently involved in the development of Raku? So please don't transfer your mistrust to these people: they have no beef in this.

But it's categorically untrue that Parrot wanted to get rid of P6 or considered P6 an afterthought or wasn't willing and able to be receptive to P6's needs, and I'm disappointed that that untruth continues to persist.

Well, maybe I don't have anything meaningful to say about this, as I have no firsthand experience in the matter. So the only thing I can do in this case, is trying to be technically correct.

I believe you had the best intentions towards Parrot and towards Perl 6 (which for some reason you keep mentioning as P6 in your replies). But, for whatever reason, it did not work out. We can keep discussing the past in that respect, but I'd rather work on the future instead.

If you really want to get the history straight, I'd suggest to write a book about it, similar to http://friendlyorangeglow.com which taught me a lot about the people developing the system I grew up with. Good and bad.

[b2gills]

The MoarVM and JVM backend supporting code was, and still are, very similar.

The Parrot backend supporting code was significantly different. So much so that to properly support Rakudo on Parrot would require a rewrite of it.

There are various reasons for that, but the main takeaway was that continuing to support Parrot made everything significantly more difficult.

(Even now the JavaScript backend code is more similar to the other two than Parrot backend code ever was.)

---

Parrot made a lot of assumptions about how objects worked.

It knew simultaneously too much about how objects worked, and not enough about how objects worked. It basically assumed that the Meta object protocol would be written in C for every object type.

MoarVM doesn't know how objects work. It gets told how they work everytime it runs. (Necessary since Raku can mutate its own object system.)

---

Rakudo on Parrot was slow. If you happened to use a Unicode character in your source file it got dog-slow.

Worse yet, Unicode was an optional feature of Parrot! WTF!!

That's not good backend for a language that has excellent Unicode support.

---

Not only did the NQP intermediate code for supporting Parrot need to be rewritten from scratch. A large portion of Parrot also needed significant work.

The main idea of Parrot was that there would be integers, numbers, strings, and other. It turns out that other was really the main category. And again Parrot was opinionated about what other even was.

It was thought that if something like integer bytecode was similar to machine-code that it would be easier to optimize it.

The problem is that something as simple as calling a function involves taking a Signature object and asking it if it accepts a given Capture object. Since even just adding two values together is a function call; doing objects fast is more important than special casing integers.

MoarVM is basically the other category first and foremost.

By that I mean if you took out the special-casing of integers, numbers, and strings from Parrot. Then redesigned the PMC code to support what Rakudo really needed, you would end up with something that is very similar to MoarVM.

The reason you might take that special-casing out is that making bytecode similar to machine-code was completely and utterly pointless. At least as far as Raku is concerned.

It doesn't matter that much if integer operations are fast if you are only ever going to run one in a row.

The reason that MoarVM isn't extremely slow is that it knows enough about objects to be able to pull them apart to only run the parts that need to be run.

---

I think that Parrot as initially designed may have been a really good fit for a Perl5+, Ruby, Python, or Lua etc; but it turns out that it really wasn't a good fit for what Perl6 eventually became.

---

Let's do a hypothetical thought experiment.

Imagine if someone did all of the work to make Parrot work again the day after it was dropped. That includes rewriting the middleware and doing enough to make the PMC part usable. So then it would have continued to work over the years.

I'm fairly certain that it would today be tied for 2nd or 3rd with JVM or JavaScript. (If not 4th.)

If the Unicode support of Parrot got better and faster I think it might be more likely to tied for 2nd.

For it to beat, or even meet MoarVM for first; it would end up needing so many changes that I'm not sure that it would even resemble what it once was.

---

I would have liked for Parrot to survive and be one of the backends that Rakudo runs on.

But dropping it was the correct move.

[chromatic]

Rakudo on Parrot was slow. If you happened to use a Unicode character in your source file it got dog-slow.

Worse yet, Unicode was an optional feature of Parrot! WTF!!

This is nonsense, and I say that as the person who spent months profiling Rakudo to figure out why it was slow.

Rakudo's parser was (maybe still is) slow because it can't optimize anything, even the <ws> token.

Adding NFG could have helped by allowing fixed-width access to normalized codepoints, but IIRC Patrick told the person who was going to implement NFG not to do it.

Seems like a pattern.

[b2gills]

You might be right about why Unicode was slow.

In which case I was right.

> So much so that to properly support Rakudo on Parrot would require a rewrite of [the middleware].

Which also means that most of the work you would have done to Parrot would needed to be reworked again afterwards.

There was `ifdefs` all over the NQP and Rakudo codebases to work around Parrot's differences. Which was annoying and error-prone.

The `ifdefs` are now mostly in NQP. And even those tend to be constrained a bit.

---

> Rakudo's parser was (maybe still is) slow because it can't optimize anything, even the <ws> token.

That is factually incorrect. There are several known optimizations that have not been implemented yet. One of which was even in STD.

Also since Raku treats regexes as code, optimizations to regular code paths can also apply to regexes. That includes optimizations to calling methods like using the <ws> token.

The main reason that they haven't been many is that the people that are competent enough and confident enough to do that work have been busy doing other things. Both their dayjobs and other optimizations or design work.

Really as far as I know, there have been next to no attempts to optimize specifically regexes and grammars since they first got to the current feature set. Certainly not in the several years where I was looking over every commit to Rakudo.