[viig99]

ML engineer here, team started as a research team, now that we have things in production and have a lot of devops, engineering work, we bifurcated into pods and work on specific bits and pieces, lot of constant fire-fighting though. Re-wrote entire stack from python to C++ threadpool async grpc (is thrift the only good threadpool server implementation available ?), deployed on openshift, used vector + influx + grafana for dashboards / internal model monitors, elastic search for loggings, lot of other tools for validation, filtering for potential training candidates etc. Right now working CI/CD for ml, during training if model finds a better model based on different validation sets, have one click deployment ready for approval etc

[dnautics]

Wow, thanks for the detail.

> Re-wrote entire stack from python to C++ threadpool async grpc

Incredible. Presumably this is for latency/performance on the inference side?

[viig99]

Yes accuracy, latency & throughput are the 3 poles we try to achieve, c++ helps with latency & throughput and helps keep the cost low.

[mlthoughts2018]

Why would c++ help with latency in comparison to say Python with numpy / numba / Cython? All the production critical “this needs to be as fast as possible stuff” I’ve ever worked on has been all Python, achieving complete speed parity with C, at a much faster development speed and with way way less boilerplate code.

[disgruntledphd2]

If you have hard constraints at inference time, then it can be much easier to tune to a time budget with C++.

Like, it's normally not worth it, but when you need it, you really need it.

[mlthoughts2018]

I definitely agree that could be a case where you want a statically compiled module that avoid any interpreted language overheads or high cost abstractions. But what would make C++ easier to write, tune, integrate or deploy in that case than using Cython to create the C++ extension for you?

[disgruntledphd2]

I dunno man, I was always against running stuff in C++ if I didn't have to, but I got over-ruled. I guess that the high availability of C++ developers helped swing the decision.

[viig99]

I personally find C++ + pybind11 vastly easier to work with, also transitioning completely to c++ from there was a pretty small leap.