Hacker News new | ask | show | jobs
by nyx 1592 days ago
This article is a lengthy preamble to an advertisement for a SaaS product, but I'm thankful you at least get to a conclusion before the evangelism begins: there's a basic pattern here, i.e. "get data from X, transform it, and put it in Y", and the tools, systems, and platforms listed are all solutions to the same basic problem.

I'm sure there's a use case and a market for this product, but I'd wager that I'm not alone among the HN audience in thinking that I'd rather have a couple shell scripts in a Docker container on my own hardware or whatever than spend a bunch of money irreversibly trapping my automations in some proprietary cloud-hosted thing where they can be held ransom come the renewal date.
3 comments
gorjusborg 1592 days ago
What saves me the most time is not "automation" but something I call "Don't waste time learning things that are likely to go away".

So yeah, I'll just leverage the skills I've leveraged over and over to make a living.

link
randito 1592 days ago
That's a really interesting point.

This seems to explain a skepticism that I have, with regards to new tools, new languages, and especially "a new [x] that will solve all your problems."

link
sidpatil 1592 days ago
https://en.m.wikipedia.org/wiki/Lindy_effect
link
Ma8ee 1592 days ago
And that basic pattern is usually called Extract, Transform and Load (ETL) and there are numerous implementations. Some respectable Open Source ones too.
link
biellls 1592 days ago
I think there's space for an open source library that can help with what you described by using just python and YAML. We originally created https://github.com/typhoon-data-org/typhoon-orchestrator to orchestrate ETL workflows, which would be a superset of the use cases you described. Our next goal is to allow deployment to AWS lambda which can be a good compromise between getting locked in with SAAS and hosting your own infrastructure.

Also check out Zappa's scheduled tasks that have a similar goal and inspired our library. We used it initially as a backbone and ran into a series of issues that forced us to write our own version of it, but depending on your goals it could be enough.

link