[paddy_m]

I would be really interested in a distributed job running system with a scheduler. I want a UI where I can set jobs via a schedule that run on one of many workers. In that same UI I want to be able to look at failed jobs and see all of the output from that job. I don't want to have to ssh over to the box where the job ran to retrieve a log file. I once had a job where we used Tidal Enterprise Scheduler which poorly implemented this. What solutions exist for this workflow in the open source world?

[yzmtf2008]

Look into airflow: https://airflow.apache.org/

[Barreljack]

Not quite distributed, but https://www.rundeck.com/open-source is pretty nifty.

[bloopernova]

Rundeck, as others have indicated, will do just that.

I've used Tidal too, it's utterly horrible and universally hated by anyone I know who has used it. What took months to set up and debug in Tidal, took us hours in Rundeck.

[nkrumm]

Also ndscheduler: https://github.com/Nextdoor/ndscheduler/

[xorcist]

Rundeck does exactly this.

If you already have something like Jenkins you could probably bend that to your will, too.

[TomMarius]

Would you pay for this? How much and what pricing scheme would you prefer?

[paddy_m]

As a service that runs on Heroku I'd pay around $25/month. that'd be $25/month * 2 (stage+prod). At the bootstrapping stage i'm at now, that's what I'd pay. If I were profitable, I'd pay up to $500 - $1000 a month and happily so, figure out market segmentation so that I'd be forced to pay that much. I would probably run around 1000 jobs a month, which seems like a number that would put me in a higher pricing range. This service would be super useful for around 60 jobs a month even. Output from the jobs would be tiny, maybe 10k at most. I would be happy if output was saved on private S3 buckets.

I guess the big point is, I'm looking to use this for regularly scheduled jobs, not just as a queuing service to spread load.

My email is in my profile if you want to get in touch.