[ramraj07]

What's the simpler solution analogous to airflow that I an deploy on the same ec2 instance as my webserver? I prefer to use the simpler tool until it becomes unwieldable, and my current stack is just an elastic beanstalk deployment that runs the webserver as well as a celery worker. This seems easy to make the site highly available (scale up and down instances easily), with managed RDS and a redis broker taking care of managing state. A scheduled task runner is the missing piece - turns out celery is just not designed for long-running tasks, and both celery and airflow seem to require that only one instance of their scheduler run at any given time. I'd much prefer a tool where the service is robust enough to handle multiple redundant schedulers so I can go home and sleep without having to bring in kubernetes just to deal with this.

Currently using some custom code to do this using a database table but totally aware of how fickle and easy to screw up this method can be, and open to suggestions.