[politelemon]

Hey wanted to give a bit of feedback.

We've found Airflow and ECS Fargate to be a great combination for running ETLs. It keeps Airflow small and dumb, and lets the Fargate containers do the heavy or complicated lifting in language of developer's choice.

We'd really appreciate if the ECS Operator could be given a bit of attention:

Running a task on FARGATE_SPOT containers is a cheap, convenient option, but it requires passing capacityProviderStrategy in. https://issues.apache.org/jira/browse/AIRFLOW-6604

Also currently the ECSOperator only shows the output logs once the task has finished (which could take hours), it'd be better if the operator could poll the Cloudwatch logs during the run rather than wait for it to finish.

---

Congratulations on the release, I'm looking forward to upgrading soon, and trying out the new features and syntax!

[dimberman]

Thank you for the feedback! I'm gonna pass that on to some AWS experts in the community.

One really nice feature of 2.0 is now the "providers (hooks, operators, etc.) are released separately from Airflow itself. So you won't need to upgrade airflow to get improved AWS operators unless there is a breaking change.

[hbcondo714]

Ditto, running Airflow on AWS ECS Fargate serverless. We did this prior to AWS announcing their Managed Workflows for Apache Airflow[1]. Do you know if and when AWS will be making Airflow 2.0 available in their managed service?

[1] https://aws.amazon.com/managed-workflows-for-apache-airflow/

[politelemon]

Have you tried Managed Workflows for Apache Airflow?

I was curious about it but the pricing page scared me off, the smallest which runs 50 DAGs is about $0.49/hr! I couldn't understand why the pricing was that way.

[dimberman]

I'm sure they will, I'm not sure on their timeline though.