[Galanwe]

I tend to agree with your view.

I attempted to port a low latency trading system from C to Rust last year, but ended up abandoning after 6 month of development.

The overall feeling that I got is that Rust made my life a dream for 80% of the codebase, the _non critical, non high performance part_.

There are tons of well written libraries, cargo is awesome, the performance is on par with what C++ would do, etc.

But the last 20% of the code base, the high performance part, was a nightmare to implement. The Rust static safety basically get in your way constantly for any kind of non obvious memory layout (self reference, etc). Dynamic safety (cells, Rcs, etc) just add too many overhead for the the critical path, and the "escape hatch" of "unsafe {}" ended up being 1000x more error prone that C.

I keep some resentment against the Rust community from this experience to be honest. I felt like instead of understanding the constraints & limitations of a fixed, microsecond time budget that I had, and trying to find solutions and be open about possible improvements, the overall trend was more in trying to defend Rust with whatever it takes.

[stouset]

At the risk of sounding like I’m trying to “defend Rust with whatever it takes”, your argument seems bizarre to me.

Even at the surface, if your desired solution is so antithetical to the type of structure that Rust tries to push you towards, you can quite literally just write this component of your code as if it was C. Just using pointers instead of borrows and tag all your functions as unsafe and… pretend you’re writing C? At that point you’ve, I think, more or less disabled 95%+ of the bits that would prohibit you from writing code exactly as you would in C. You don’t get the benefits of Rust for that part, but at least you get them for the remaining 80%.

Or you can very literally just write the “hard part” in C and call it from Rust. You might have to make sure you can’t panic across language boundaries, but other than that the C interop is just about the best I’ve seen from any language.

I also don’t entirely understand how unsafe can be 1,000x more (or even 1.5x more) error prone than C. But I’d love to hear how. The only “trick” to unsafe is that you should aim for your unsafe blocks to be “unit-safe”. Meaning they might do something unsafe inside, but from the outside looking at it as a black box, the unit functionality of it should be safe. I don’t think the docs do a good enough job here of encouraging that style of design. You can violate this guideline, but doing so without sufficient care is quite likely to result in bugs. But of course if you did the same approach in C, you’d have a similar outcome.

The only real way I can reconcile your points is if the performance-critical bits that seriously impact the design of your program are scattered uniformly and don’t have anything resembling clean boundaries. I suppose that’s a real possibility but it does seem very foreign to me.

[Galanwe]

Yeah see, that is what I thought at first as well.

I sort of imagined that I could get the best of both world, and just "unsafe { <C style code> }" my way out for performance critical things.

But the thing is, the static safety boundaries of Rust allow the compiler to make much, much tighter assumptions than C & C++, especially around aliasing rules, un initialized memory, and moves.

When you relax these boundaries with "unsafe {}", you don't enter "C world", you enter the litteral gates of hell where any innocent temporary cast can throw you in a random load/store reordering bug.

[iudqnolq]

Do you mean innocent temporary cast from a pointer to a reference? Cause yeah, those aren't innocent. And that's a newer realization and there were serious documentation issues around it.

[stouset]

Thanks for the reply. I think if you combine unsafe with raw pointers instead of borrows you relax the rules enough to avoid this. But in this specific corner of things, I haven’t had direct experience so you may very well be right.

There are still some Rust-specific details you would still need to handle—as you mentioned, uninitialized memory—but for that one specifically I haven’t found MaybeUninit to be particularly cumbersome.

[twic]

> Dynamic safety (cells, Rcs, etc) just add too many overhead for the the critical path

A low latency trading system can't have Rc, or any C equivalent of it, in the critical path, because it does allocation. More generally, your story doesn't ring true to me, because the performance-critical parts of a low latency trading system have to be so simple that there isn't scope for any of the tricky bits you talk about, they have to be braindead simple loops over simple data structures.

One of my colleagues wrote a low latency trading system in Rust. He had to learn Rust to do it, already knew C, and today thinks this was the right decision.

[Galanwe]

> your story doesn't ring true to me, because the performance-critical parts of a low latency trading system have to be so simple

Hu, so it doesn't ring true because... you feel like it's too complicated from what you imagined? That's a first.

> that there isn't scope for any of the tricky bits you talk about, they have to be braindead simple loops over simple data structures

That is far from the reality of low latency trading systems. Let's just look at the very first building blocks:

1) You will need some form of userspace packet polling, along with its associated multicast A/B line arbitration. That alone is not "simple braindead loops", and we're just talking about getting 1 UDP datagram here.

2) You will need some form of packet queueing before decoding, so that you do not drop packets under bursts. That most likely means some variant of lock free ring buffer in shared memory. Far from "a bunch of brain dead loops" as well.

3) You will need to decode packets and maintain an order book state. That mostly implies a purpose built b+tree. If you've implemented any of those, you will know it's also far from simple.

4) You will have to compute some indicators based on order book state, which most likely will lead you to some form of linear algebra.

5) You will need to disseminate such indicators to perform some simple close formula variant of portfolio optimization. More linear algebra.

6) etc.

I'm happy your colleague that he managed to do all that without struggling, that was not my experience, that's all. But to say that it's "just braindead loops over simple datastructures" is either a sign he didn't do a _real_ low latency system, or he just was somehow spared from the real underlying complexity.

[baq]

This is a common thing to say for folks who are really good at low level C/C++. You guys know too much, seen horrors no one should have ever seen and came out alive on the other side.

The problem is, the things you needed to do to stay alive in this environment are antipatterns in Rust and you’ve learned the hard way Rust actively opposes being fed Rust antipatterns.

Self referential data structures are some of the hardest problems in CS to get right. There are patterns that work with Rust which replace pointers with eg generational indexes. You’ll find people do the same thing in C++ once they get burnt one time too many. I think you’d have a very different experience if you approached your system with an ECS framework, either off the shelf or roll your own.

[insanitybit]

> and the "escape hatch" of "unsafe {}" ended up being 1000x more error prone that C.

IMO this is the biggest issue right now. `unsafe` needs some refinement, certainly.

> the overall trend was more in trying to defend Rust with whatever it takes.

This can sometimes be the case, unfortunately. But fwiw I think a lot of us in the community agree entirely that there's room to improve in the unsafe world.