|
>and I was baffled why something so trivial as Airflow was being sold as a critically important piece of infrastructure. https://news.ycombinator.com/item?id=9224 >Something as simple as "job A must run after job B and job C, but if it doesn't start by 2am, wake up team X. If it doesn't finish by 4am, wake up team Y" isn't Airflow's problem, it's your problem. I guess that's one approach to job security. And why not make data egress manual too? Why transfer data through the network, when you can print them, mail the papers, and type them back in? Data input is not the computer's problem, it's your problem! >Something as simple as "job A must run after job B and job C, but if it doesn't start by 2am, wake up team X. If it doesn't finish by 4am, wake up team Y" isn't Airflow's problem, it's your problem. "What's the overall trend for job D's finish time, what is the main reason for that?" isn't Airflow's problem, it's your problem. "What jobs are on the critical path for job E?" isn't Airflow's problem, it's your problem. "Job F failed for date T and then recursively restart everything that uses its results for date T" isn't Airflow's problem, it's your problem. The whole idea of writing programs is making things automatable. That is, making them the computer's problem, not our problem. We get the higher level problem of writing the automation once, and fixing any bugs in our code, then we get to enjoy putting it to work for us... |