Hacker News new | ask | show | jobs
by bdd8f1df777b 630 days ago
The biggest use case (that I am aware of) of GIL-less Python is for parallel feeding data into ML model training.

* PyTorch currently uses `multiprocessing` for that, but it is fraught with bugs and with less than ideal performance, which is sorely needed for ML training (it can starve the GPU).

* Tensorflow just discards Python for data loading. Its data loaders are actually in C++ so it has no performance problems. But it is so inflexible that it is always painful for me to load data in TF.

Given how hot ML is, and how Python is currently the major language for ML, it makes sense for them to optimize for this.