[hermitdev]

Your experience largely mirrors my own. At a previous employer, I had to make some changes to a process that was importing data from a vendor. Typical straight-forward ETL, right? Not even a lot of data, like 20-30 records daily.

The process? Load the data from CSV to JSON, ship the JSON off to Azure. Pull the data back from Azure, check to see if it's been processed, apply the change on premise. If failed, reschedule for later. Multiple processes, multiple scheduled jobs, on-call alerts, etc, etc. The whole thing could have been replaced with maybe 50 lines of Python. Instead, it was probably around 10k lines of C# and dependency on a 3rd party ETL tool. It was a fucking mess. Worse, yet, I wasn't allowed to fix it.

[pmkiwi]

This reminds me a colleague creating a Python class like this

import os

class Bla(): def __init__(self):

        self.var1 = os.env["VARIABLE1"]

        self.var2 = os.env["VARIABLE2"]

        # ...

        self.varN = os.env["VARIABLEN"]

    def get_variables(self):
        return self.var1, self.var2, ..., self.varN

They took 2 months to write an insanely complcated code to just copy few objects from an S3 bucket to another... And because the Lambda was timing out, they set the timeout to 15 minutes, leading to our Lambda costs skyrocketing because the function was failing/retrying all the time for some objects.

I was allowed to fix the timeout issue to save cost but I was forbidden to fix the code itself because our manager said "Well, it works so let's move on".

6 months later, on a Tuesday morning, bored, I decided it was enough so I rewrote this bloody Lambda in ~90 lines with a proper handler, retries, logging statements, etc ... in a couple of hours.

What did my manager say? "Good work but you should have taught them rather than doing it yourself".

[tempest_]

He is right though, gotta stop the problem at it's source.

Often however that is a far larger/difficult task and it is easier and better for ones sanity to just fix the things that affect you directly.

[merrywhether]

To be slightly pedantic, one might call this situation over-architected rather than being purely over-engineered. I would agree that over-architected solutions (regardless of the internal engineering quality) are definitely hard to alter due to the many disconnected disparate parts, not to mention any human/political boundaries that have grown up around the implementation.

Following along that train of thought, under-architected solutions are often great to update because you get to make logical cleavages that are informed by time spent in actual production use, giving you a much better basis for any decisions.

[yongjik]

Amen. And the worst part? Can't even fix it easily, because they all depend on each other. Someone is getting the data in json format and depends on the filename being exactly YYYYMMDD in a specific directory of a specific server. Some horrible open source framework can only handle data at 200 requests/sec (and of course every row is a request) and you have a distributed asynchronous queue plus rate-limiter to make it work. Yada yada.