Hacker News new | ask | show | jobs
by righthand 606 days ago
I worked for a company and implemented testing suites into our infrastructure. My team and I spent time meeting with all the various stakeholders, FE, BE, Auth, Infra, etc. Put the solution in place and built out the dev tools needed, held constant training with teams, answered questions when called out in org meetings. Since we were building two docker images (1 the actual image and 1 for proxying network connections for tests) During development I asked 100 times that this would not increase cost. Everyone assured me we did not have to worry because that is all negotiated yearly.

At the end of the day we had this weird error where the test suites would randomly fail. Always a different test, different time of day, different engineer running into the issue. This caused “bad days” for all the feature developers.

I kept investigating and pushing back that it wasn’t my implementation. Lo and behold months later it was revealed that any subsequent PR would cancel the docker image built for the active PR. The tests would fail because the image was getting trashed. The reason this kept happening is that the QA env was not actually setup to mirror the production env, as a cost saving measure done much much much before my time.

However since engineering and infrastructure refused to address the issue months ago, the dev org had built up ill will against me. Everyone blamed me and instead of sitting down and still fixing the issue to have a parallel environment to our production env, they laid me off, and removed all my work.

I still have friends that work there and they still fight daily about testing deployments and rolling back because they removed the testing in place.

2 comments

> the QA env was not actually setup to mirror the production env

As senior QA, this alone is going to end me one day.

Ah yes, the “you touched it last” or the venerable “you brought us the bad news” schools of management.

I also love the “we’ll do the meta-work that improves work velocity after the work is finished. Not before! We’re too busy for that now.”

Yep, we’ll optimize it later infra told me. I hear it still at my current employer. Optimize it later means you’re touching on a long ignored piece of negative tech debt.
"There is no later." is my new mantra.
“I’ll just use a different provider that isn’t our inhouse over-engineered AWS yaml config platform.” Is mine.