There are some other lessons too, especially regarding Haskell. Basically the sad truth is that if you do not benchmark literally every single line, you can suddenly get MUCH worse performance than expected. Sometimes GHC's strictness analysis does not discover that it can unbox some things and you land in a boxing-unboxing loop or you hit some of optimization bugs (during the work on performance we've reported over 10 GHC bugs and even implemented our custom graph memory manager). So one of the biggest problems that we encountered earlier was that actually the GHC performance is MUCH harder to predict than you think and you have to take extra care of it while building your software. Of course using Haskell pays off in other areas, but performance is tricky.
Thanks for the details. I would love to read still more about it to learn from the experience. E.g., What would be examples of the boxing-unboxing loop. All you have described seems worthy of a more detailed blog article in its own right. :-) Thanks.
To be blunt, performance is much more important than you think, even when building a proof of concept. We used the PoC code for much longer than was initially intended, and had we thought about performance from the start we might’ve saved quite a bit of work!