Hacker News new | ask | show | jobs
by dominotw 1416 days ago
Just put everything in a warehouse( via fivetran or some such thing) and just use DBT.

Use airflow as cron runner for dbt.

If you don't need realtime metrics, this formula works way better than convoluted airflow dags.

2 comments

We have had a great experience scheduling Meltano/dbt inside Orchest for our Metabase dashboards. As a pattern, combining these declarative/configuration CLI tools with a flexible orchestration layer (Orchest can run any containerized task, it will containerize transparently for you) really shines.
If you need cron, use cron or a cronjob pod. Airflow is a poor cron scheduler at any sort of scale.
yea for sure. Not sure what a good cron runner with web ui with logs is, jenkins?