Hacker News new | ask | show | jobs
by luckydata 3565 days ago
It's the tooling, not the size of the data. Using "big data ecosystem" tools allows you to use all kinds of useful things like Airflow for pipeline processing, Presto to query the data, Spark for enrichment and machine learning etc... all of that without moving the data, which simplifies greatly metadata management which has to be done if you're serious about things like data provenance and quality.
2 comments
xorgar831 3565 days ago
A SQL db + Tableau are vastly more powerful and mature than those tools, they just can't do "big data", that's all.
link
luckydata 3563 days ago
the data preparation work involved in doing anything trivial with that setup is mind boggling. I'm going to assume that this very cavalier assumption comes from a place of relatively little knowledge. Do some of this and you'll realize very soon you'd rather shoot yourself than doing complex data prep on constantly evolving schema using schema-on-write tools.
link
qwertyuiop924 3565 days ago
...and how often do you need to do all that to a dataset?
link