Hacker News new | ask | show | jobs
by shalabhc 1198 days ago
(author here)

I agree it would be fantastic to have sub second deploys! Doing this for user specified Python environments is challenging in different ways than doing it for a JS SDK like Workers. Note that just provisioning a GitHub runner takes about 10s, before any deploy code even starts up. In theory we could rsync the code directly to the code server and reload it. But any of these options require bigger architectural changes.

Also you can already have a local environment setup for faster iteration and use this for the shared environments with other developers, where the speedup is still great to have.

1 comments

> Doing this for user specified Python environments

Who is using this/asking for this/why?

dagster.io is an open source Python library for building data pipelines. When using Dagster your project may depend on other Python data analysis and warehouse libraries like pandas, snowflake, pyspark and so on. Each org's requirements for their data orchestration will be different. When executing the pipelines in Dagster Cloud, we need to replicate their Python environment.