Hacker News new | ask | show | jobs
by kasey_junk 2817 days ago
Maybe. But I’ve been developing golang full time in high scale concurrency environments for 4 years and working with a team of similar people. It’s an opinion that is near universally shared on that team.

At high concurrency levels almost everything abandons standard golang concurrency patterns and tools.
1 comments
tandr 2817 days ago
Would you be so kind and explain what term "high concurrency levels" implies?

thank you in advance.

link
kasey_junk 2817 days ago
Consumer facing systems. System wide throughput between 6-12 million QPS (daily low/high) (query body average size is 1.5KB). Each server tops at ~130K QPS. On a system with 2 1 GB NICs we pop the NIC. On a 10G we pop the CPU.

Current bottleneck is the golang http/net libs. Would likely need to rewrite it from the NIC up to do better.

link
wahern 2813 days ago
That's an issue with the http/net libraries, not the concurrency model.

At really high throughout you can run into into issues with the kernel's networking and driver stack. I've encountered situations with my own homegrown event libraries (mostly C or Lua+C; I've never used Go) that were bottlenecked in the kernel. I've also seen issues that were fundamentally related to the use of poor buffering and processing pipeline strategies that resulted in horrible performance. For example, I can get an order of magnitude greater streaming throughput using my own protocol and framing implementations than when using ffmpeg's, though I use ffmpeg's codecs and a non-blocking I/O model in both cases, all in C. And that's because of how I structured the flow of data through my processing pipeline.

There's is no general model of concurrency that can solve that, and I've never seen any model that was easier in the abstract to tweak than any others. Those are implementation issues.

link
apta 2816 days ago
Do you think the JVM would have faired better at this? Otherwise, perhaps something like Rust would be a better fit.
link
kasey_junk 2813 days ago
I don’t know if it would have been holistically better, golang has lots of advantages.

But the concurrency would have been more straight forward on the JVM because the language allows for more choices & their are lots of options that get you there.

link