|
|
|
|
|
|
by dilly_li
2133 days ago
|
|
|
Is there a workflow tool that’s designed with micro service in mind? My particular use case:
- several workers process the data on the workers’ local threads
- several workers serve as relays to interface with external third party services, hold all the necessary credentials, and conduct cron-like checking.
- the ETL tool doesn’t directly provision these workers. The second point is part of the reason why we don’t want to use Airflow’s k8s operator. But it doesn’t seem like there is a better option in terms commonly used and robustness. So we are leaning towards write some custom operators and sensors to make Airflow more friendly to micro services. Thought? |
|
|