Hacker News new | ask | show | jobs
by ketozhang 1692 days ago
Look into Dask if you are attempting to process an entire data table that's larger than memory.

For your last paragraph, you're conflating the need to share code with the need to build a robust scalable service. Most research code are only needed for the paper and rarely touched again.