[thenrich99]

Amazon Data Pipeline and Cronally solve different problems.

ADP is a complex service designed to transport and process data between AWS products in a time-sensitive manner. Minimum intervals of 15 minutes and the inability to run jobs on a set of existing machines (i.e. running a shell command pipeline requires a separate instance launched for each execution) doesn't support a number of use-cases.

Cronally is a simple platform that augments an existing AWS cluster by allowing those same machines (or others) to perform work, while handling locks transparently and supporting resolution down to the minute.