Hacker News new | ask | show | jobs
by 6LLvveMx2koXfwn 448 days ago
We deploy on K8s in OpenStack from a scheduled GitHub Actions pipeline which aggregates DAGs into a new container build based on hashes of hashes. This works well with almost no intervention.

WRT your 1, above any DAG output to stdout/err is available via the logs tab from the graph view of the individual tasks. Almost all our DAGs leverage on the PythonOperator though, not sure if that standardises this for us and your experience is muddied by more complexity than we currently have?

WRT 2. we generate an uber requirements.txt running pyreqs from the pipeline and install everything in the container automatically. Again no issues currently - although we do need to manually add the installation of test libraries to the pipeline job as for some reason auto-discovery is flakier for unit-tests frameworks.