[generalk]

This is the way.

My previous company had a HUGE problem with Devs cowboying off and doing whatever and dumping it on the Ops team at the last minute.

One of the biggest (but for damn sure not the last) issues was a dev who designed and built an entire new product around a MongoDB database, which wasn't something we had in production, and something he didn't mention during the months of development and demos to stakeholders. Week before the launch date he hits up our Ops folks to get production set up.

Ops was calm and collected about the whole thing. "We don't have MongoDB in production. Are you volunteering to learn how to correctly install it, write monitors for alerting, be paged with issues, figure out backups and how to ensure our data stays safe, secure, and available? You're not? Then get the [redacted] out and rewrite your app. Yes it will affect the ship date, and yes it's your fault."

I'd love to say we used that opportunity to shore up our processes involving kicking off new applications and including Ops folks in from day one, but that took years more.

[time0ut]

Something similar happened at my company like 5 years ago.

A developer was tasked with adding a major new feature to one of our older monoliths. He added MongoDB as a dependency. The application already had a well managed Oracle database. Nothing about the feature required MongoDB.

When it came time to go to production, the DBA and ops teams responded similarly to how you did. I wish I could say sanity prevailed, but the business mumbled something about contractually obligated release dates and forced it through to production. Pretty sure it is still there rotting away.

I've worked mostly on the app side of things and this sort of thing just makes me shake my head.

[random_kris]

well at the end of the day you managed to ship it? Did it cause any big problems down the line? It seems the biggest problem is that it is rotting away somewhere, which to me means that it is working without need to do much care on it.

If they listened to your DBA/ops guys no value would be gettig shipped ;)

[time0ut]

I don't know of any big problems other than the unnecessary cost. I agree meeting the needs of the company is king, but it was just a lot of unnecessary complexity because a dev wanted to put MongoDB on their resume. Could have been avoided by talking to the rest of the team early on. Of course, they would not have liked the answer of just creating a new table in boring old Oracle.

[Aeolun]

To be fair, when forced to choose between Oracle and MongoDB I’d also have a serious dilemma.

[KronisLV]

> I agree meeting the needs of the company is king, but it was just a lot of unnecessary complexity because a dev wanted to put MongoDB on their resume.

Counterpoint: the dev is doing this to remain employable, so that they can ensure higher success in the future for themselves.

Their goals simply don't align those of ops and are at best parallel with those of the company as a whole - of course it's to be expected that they'll attempt to prioritize their own when there's a lack of governance and oversight within the company.

It's something that i've noticed more and more, yet is something that noone really talks about - people wanting to use bleeding edge technologies just because they're at the top of their hype curve: wanting to implement microservices when they're just maintaining monoliths and there's no need for them.

Personally, i'm an advocate of both microservices (or at the very least modular monoliths), containers and many of the new technologies, with the exception that i've initially tried all of those out in personal projects in the evenings and weekends. Yet what is the person who doesn't code outside of work supposed to do to remain employable? Would you expect a doctor to practice new types of surgery in their own time? Actually, why don't companies fund a week every few months for their developers to upskill themselves? Just a bit of time that's treated like a vacation, but during which they're expected to hack together prototypes etc.? Clearly most companies out there don't do greenfield or pilot projects, so something like this could help.

I don't think i have any good answers for this, but it definitely deserves more consideration!

[oblio]

> Ops was calm and collected about the whole thing. "We don't have MongoDB in production. Are you volunteering to learn how to correctly install it, write monitors for alerting, be paged with issues, figure out backups and how to ensure our data stays safe, secure, and available? You're not? Then get the [redacted] out and rewrite your app. Yes it will affect the ship date, and yes it's your fault."

Love the shoot-down!

[davidgerard]

Ops here. The threat of 3am phone calls does wonders, in my experience.

If it turns out it was product owner pressure, the product owner gets a call too. Possibly first.

[Aeolun]

So, you could have delayed the app by the same amount but now have a mongo environment for production as well?

Seems a bit of a waste to rewrite the app instead.

Not that I would recommend Mongo anywhere, production or dev, but it would apply for any other technology for which this happened.

[generalk]

  > So, you could have delayed the app by the same amount 
  > but now have a mongo environment for production as well?

No, we couldn't have. Not just because we didn't want MongoDB, which at the time was notorious for data loss, but because our ops team didn't have the capacity at that point in their schedule or team size to handle it. Maybe had we discussed at the beginning of the project plans could have been made or altered, but we didn't and so they couldn't.

  > Seems a bit of a waste to rewrite the app instead.

The responsible dev took the time necessary to rewrite the data layer to better reflect the needs of the application.

Is what I wish had happened. Instead the developer jammed the huge JSON blobs into a column on an MSSQL table and changed a few lines. lolsob.

[jimbokun]

> Instead the developer jammed the huge JSON blobs into a column on an MSSQL table and changed a few lines.

Sounds like quickest way to deliver value to the customer. As described, was far too late in the process to worry about deploying with a clean, extensible architecture.

A reasonable amount of technical debt in order to ship in the timeframe available.

[rswail]

Depends on your definition of "reasonable". You now can't leverage your DBA's skills to optimize queries, because you're using the RDBMS as a key/value store.

You're misusing a tool because you didn't do the correct application design in the first place.

NoSQL has it's place, mostly in the trash. Lazy key/value stores (which is all that NoSQL is) throw away all the benefits of relational logic for a glorified combination of a file system and grep.

That's not "delivering value to a customer", that's delivering crap.

Standard "Agile" response. It was only "far too late in the process" due to a complete lack of process, oversight, product quality ownership and capabilities.

If nothing else, that developer should be "counselled" as should the PO, the Scrum Master and anyone else involved that allowed the situation to occur.

And the ongoing capex and opex for the additional unbudgeted support should be pushed back on the PO as a requirement to fix.

[kazen44]

except that shipping something with semi-broken infrastructure leads to losses down the line.

What if your mongodb database drops its data and now you have production impact? Are those losses calculated while making these decisions during development.

[Aeolun]

> because our ops team didn't have the capacity at that point in their schedule or team size to handle it

Lol, I get your point, but that was also true for the dev organisation. Hence what you ended up with.

I doubt the needs of the application included a rewrite in MSSQL.