|
|
|
|
|
|
by jeshin
1440 days ago
|
|
|
dicts are just a little too easy to use. You just smear it down, pass it around, and you're in business. If you really want to shoot yourself in the foot, also modify its structure here and there along the way, it's just so convenient. Who needs all that hassle of declaring a data class for each little thing? It took me a little too long to realize that a data class represents a contract about the structure of your data, meaning that no matter how many calls deep you are passing it around, you will always know its structure without having to trace it back to the origin, and that's a powerful thing. |
|
|
Highlights included:
* DIY “json” logging function that did some obscene string concat work every time it was called and abused global vars; it also didn’t output valid JSON. Suggestions to just use a normal logging library were aggressively disregarded.
* dozens of functions, all of which indirectly mutated this extremely nested dictionary of data. They all had slightly different names, and none of them took this dictionary as a parameter, they just abused global vars. All of them would do these insane checks to ensure that the specific keys they were looking for existed
* none of it was in git properly; the 2 data engineers writing it passed the code back and forth using a google drive.
* instead of importing functions from the Python files they wrote, they’d invoke the functions by shelling out, calling Python <other file.py>, string interpolating the values and then waiting for completion by *waiting for a file of a specific name to be written into the file system.
Oh yeah and when they decided they wanted parallelism, instead of doing the sane thing and using something like joblib or multiprocessing to make stuff easy, they’d just shell out and invoke more Python processes via xargs…