|
|
|
|
|
|
by icedchai
6 days ago
|
|
|
These sorts of odd projects are relatively common. A few years ago I was brought on near the end of a data engineering project where somebody had decided they needed multiple databases, a crap load of JSON exports, and dozens of python, R, and shell scripts running inside some job orchestrator to support what amounted to a few megabytes of data being processed each day. Maybe 5 megabytes max. There wasn't even a lot of transformation going on. It was just... strange. I witnessed some true eldritch horrors like Python calling R calling a shell script that called the mysql client, which wrote data to a temporary file that was eventually read by the great-grand-parent python script. |
|
|