[simag]

Completed a full rewrite of many components of the Kraken.com backend in about 4 years.

The new system is around 1.5M loc of Rust. There was no serious alternative to rewriting, sometimes you find yourself in a corner and need to fix issues, and pay the price.

I wrote about it 3 years ago here https://blog.kraken.com/product/engineering/oxidizing-kraken...

Everything in that blog post still rings true and hindsight is that it were were right. But it was a massive grind and required extreme dedication to get it done, for a variety of reasons that work was very taxing.

We also didn't stop feature development and kept the two systems running concurrently (which explains why it took so long, also growing and training a new team 10x the size took time, so there are many factors).

I'm also against rewrites if I can help it, but reality is complex and sometimes we can't help it. Now however, since we removed the last pieces of legacy that were preventing larger DB schema changes (or required massive, unreasonable changes to the legacy systems), we've been shipping faster and easier than ever and caught-up on a lot of the accumulated backlog, including some of the more ambitious projects that were unthinkable in the legacy systems due to limitations.

[SenorKimchi]

Huge fan of Kraken.

Looking back, is there anything that you would have done differently? I find that half or more of the rewrites that I have dealt with have been driven by all the wrong motivations. You get inevitable turnover and at some point people dislike code that they didn't write themselves and push for a rewrite, maybe changing the stack to something trendy, justifying it with thin arguments. Once the rewrite starts the company ends up treading water for years while incurring a ton of costs. For me, I think only 1 rewrite that I was part of was a good decision in my 15 years in tech. If I could go back in time, I think I would kill all rewrite discussions the moment that someone first whispers the idea.

How did you guys enjoy switching to Rust? I assume the safety and performance benefits for the trading system are a huge plus (didn't Kraken trading go down for an entire week a few years ago?). Did you also rewrite the webapp backend in Rust as well? How has staffing and budgeting been affected? I would assume that the supply of Rust developers is much lower unless you train them in house. Rust sounds fun, but I can't imagine trying to justify a rewrite of a legacy system, a major tech stack change, and training/building a new team all at the same time.

Sorry for the onslaught on questions. The "rewrite it in rust" fever has spread to my work and I'm fighting myself on how to respond.

[simag]

With hindsight, considering the cards we were dealt, there's not much I would have done differently. If I had known better before, I would have ensured stronger buy-in because after a while our internal stakeholders were often pushing back on the effort, and that led to concessions where throw away code in the legacy systems was built even for weak business outcomes.

Overall I share your concerns. Having the right reasons to rewrite is key. I believe this blog[1] about software as theory building does a great job at describing the challenge with software gardening, and the times where a rewrite is the solution are few. Even then, it's critical to handle the rewrite in ways that can work - in our case, we chose to progressively eat the legacy software without making major changes when we could avoid them. The legacy software we had was mostly the results of one man heroics and traded off performance and availability for correctness and security. It also was designed to be maintained by a small group. Solid choices if you are early Kraken - but as many successful startups, we were victim of our success and we needed it all.

When it became clear that we had to rewrite the stack (the 2017 3-days shutdown happened just before that realization), those in charge at the time decided to experiment with Rust. It was a crazy bet in early 2018, it was still Rust 2015, no NLL, no async, far rougher ecosystem. The fact that it became successful enough to warrant pursuing a full rewrite is to credit on some lucky hires who made it a success.

In that regard, Rust was a very strong talent magnet. In my experience, having hired 200+ Rust engineers over the last 5 years, there are a few kind of engineers attracted to Rust: (1) some just like shiny things/hype, (2) some are perfectionists and never complete a project, (3) some just are doers who have found that Rust is a particularly effective language.

Overall, Rust has been a great to hire for. Many engineers out there want to use Rust, even if it's their 1st Rust professional experience. We were also known in the Rust community for hiring for full time Rust, probably also the place currently with the highest density of Rust talent (there are massive companies with more Rust devs, but smaller % overall). Budget wise, our Rust engineers are not paid particularly better than other engineers in the company, but the compensations at Kraken are generally in the higher tier.

At the risk of sounding boastful, in my experience Rust is reasonably easy to learn for experienced/strong developers (we have some very young outstanding Rust devs as well, most of the time they learned before joining). Average developers struggle and may never become productive. Again, we have an engineering excellence culture so it is okay for us, YMMV.

Re scope, yes we use Rust for everything in the backend, including CRUD type of work like Web APIs. We've found we're at least as productive than other languages (Go / Java+Spring / Ruby / PHP) while having far fewer incidents, and easier maintenance / cheaper KTLO. Rust's ability for reuse is excellent which means that there are very strong network effects when having more services in Rust, including the Web layer.

A nice "side effect" of a full Rust stack is that our p99.9 latency internally is usually stable around 3-4ms for most operations, even though multiple services are involved. That's coming from a much higher baseline with much more deviation across operations providing the same functionality (60-100ms).

Regarding your own rewrite discussions, you're not going to be convinced by a post on HN, I'll just say that I am very reluctant to even think working at a workplace that doesn't predominantly uses Rust. I've been in the industry for 20+ years, across many stacks and there's a before and an after Rust for me. It has been a super power and made our life easier. It makes it easier to model business problems thanks to algebraic data types and their usage for error handling (versus inheritance), traits allow to abstract behavior better than OOP-style interfaces, the absence of data races is a game changer for multi-threaded code, dependency management is trivial, the ecosystem is rich and things work well. A lot of these are properties found in other languages but no other has the same full package and is on its way to become mainstream.

[1] https://www.baldurbjarnason.com/2022/theory-building/

[swah]

That's one of the best ads I've ever seen ;)

What about the excessive typing - in practice do you get used to it? Are you allowing Copilot and such inside the business?

[simag]

If by excessive typing you mean complex trait bounds and the like, it's really something isolated to the most generic libraries, and reasonably rare. Most application code is simply typed - take a struct or an enum, return another. So it's not been a problem.

We don't use Copilot or any tool that sends our source code because of our security posture, defense in depth, etc. If and when there will be an offering that lets us run the server component and provides significant productivity improvements, we will subscribe.