[Manishearth]

> Try using Go or Rust (love both, x2 for Go) to allocate say a hundred GB of memory for some huge/fast in-memory data processing. Let me know how far you get.

There is no fundamental reason why this should be slower or harder in Rust. Rust generally compiles down to more or less the same code C does.

There are reasons why this could be slower in Go, but it really depends on what program you're writing, so it might even just work fine. If you don't hit the GC, for example (and Go gives you ample opportunities to not hit the GC), data processing should be quite fast. But it depends.

I'd love to hear real-world experiences with such systems in Go.

[courtf]

We have a few Go processes with high memory usage. For one in particular, while it's been higher in the past (~150GB), we're sitting at 40-80GB per node right now.

The busiest node traffic-wise had average GC time over the past 20min of 3.4ms every 54.5s. 95th percentile on GC time is 6.82ms.

That node is sitting at 36GB in-use right now, and has allocated (and freed) an additional 661GB over the past 20min.

Can't really speak to how fast this is vs other environments, but it's smooth sailing overall. /shrug

[Manishearth]

That sounds much better than the Java stories I've heard, which makes sense since Go is better at avoiding the heap.

No idea how it compares with others; and not sure if it is representative, but to me that sounds pretty decent.