[villasv]

I have found that using KillMode=mixed is very useful when running Airflow with Celery workers. It allows the system to shudown gracefully coupled with TimeoutStopSec, so the workers will stop receiving new jobs but will finish their current jobs before exiting which is nice for auto scaling or spot instances on AWS (coupled with EC2 lifecycle hooks).

[1996]

Interesting. Why do you care about graceful shutdown?

Why not kill ASAP and restart?

Do you have problems with load booming up sometimes?

[akramar]

In some cases we need to do an update-deploy-restart while a DAG is still running (not even the one being updated). Then several minutes or hours later the child processes raise a segfault and the jobs they were working on fail, requiring restarting any of those jobs. I imagine a graceful shutdown would allow the job to finish up and the DAG to continue with the remaining jobs.