Hacker News new | ask | show | jobs
by ealready_value 6 days ago
I've not gotten a straight answer. I assume it is a pet project kind of situation, or trying to justify the data warehouse project as a whole, but I really don't know the real driver to do this.
1 comments

These sorts of odd projects are relatively common. A few years ago I was brought on near the end of a data engineering project where somebody had decided they needed multiple databases, a crap load of JSON exports, and dozens of python, R, and shell scripts running inside some job orchestrator to support what amounted to a few megabytes of data being processed each day. Maybe 5 megabytes max.

There wasn't even a lot of transformation going on. It was just... strange. I witnessed some true eldritch horrors like Python calling R calling a shell script that called the mysql client, which wrote data to a temporary file that was eventually read by the great-grand-parent python script.