Hacker News new | ask | show | jobs
by richardw 483 days ago
I’ve assumed that it’s partly because the company has done a lot of HFT, which is very focused on performance. But I’m not an expert in either.
1 comments

Indeed, the blog mentioned in the other comment showed part of 3FS code was completed at least since 2019, when this was still a project of the quant funds. In HFT, you tend to dogfood a lot of the things to achieve low latency, high performance, sometimes just because HFT system just need to do one specific thing, and those off the shelf stuff usually cater for a lot wider scenarios where HFT doesn't really care about. Here you see similar case which they focus specifically on loading large amount of data during training, and implement that to the extreme.