Ask HN: The Worst Engineer I Know Is Demanding We Start Using Celery (Python)

[theorlandog]

I work with an engineer who consistently advocates for terrible architecture and design patterns with a ton of confidence. Every time one of his ideas is implemented it turns into a flaming bag of complexity dog shit, and everyone else is left holding the bag to clean up.

His new tirade is demanding we start using celery to solve every problem we have. I instinctively want to dismiss him out of turn given his previous track record of thinking himself into a pretzel, however, I don't want to fall into a trap myself of negative thinking.

The way I see it, we would already be sending async job messages to a durable queue like SQS. It seems extremely trivial to me to run a service that polls the queue for new messages and dispatches the jobs on its own from there (as I have a done with great success a number of times in other environments). What is the benefit of pulling in all the complexity of celery for these simple asynchronous tasks?

[viraptor]

> It seems extremely trivial to me to run a service that polls the queue for new messages and dispatches the jobs on its own from there

Are you handling retries, timeouts, disconnected clients, logging failures, dealing with perma-failing jobs, priorities, execution metrics, etc.?

You may or may not be interested in those features, but that's why projects like celery exist. It's not "extremely trivial" to replicate them all. If you have only extremely simple tasks that you don't critically depend on, that's fine to use simple solutions instead.

If you want a constructive discussion about things like that, the questions are: what problem are we trying to solve, what features we need today/soon, how long will the implementation take, what is the risk in not changing anything.

> Every time one of his ideas is implemented it turns into a flaming bag of complexity dog shit

How sure are you it's due to the idea itself rather than due to the implementation? If it really is due to the ideas and it repeated, that's a management problem more than that person problem. Maybe start chatting to them?

[odie5533]

Celery sounds like the right choice here.

[Tinos]

Sounds like your company/team have some serious communication issues. These things do happen in all companies regardless of size - but from what I understand, this is a repeat event.

Mentioning these sorts of concerns with other people in the company (especially higher-ups) is crucial. They can offer you the guidance on how to handle the situation far more effectively than some strangers on a forum.

I have had many instances of these sort of situations. Half the time you just suck it up and do it (if the person is more senior or it's a topic you're not so confident about) and the other half you can usually find some other strategy like meeting in the middle.

Maybe Celery is the best option? Have a team-wide meeting to discuss his side of the story. Sure, he might be one to just advocate constantly, but give him his pedestal to speak. If his arguments are shaky, he will step down from the pedestal on his own when he realises that this decision is inefficient and unnecessary. If he's not the founder/CEO then talk to the founder/CEO or anyone above this guy so that they can help you put him in his place.

Sorry I couldn't be of technical help but I hope this feedback will help tackle your team's root problem - communication.

[theorlandog]

You are hitting the nail on the head with the communication issues. Truth be told, I am more senior and the rest of the leadership team understands that this individual needs a lot of management and shouldn't be allowed to implement stuff on their own.

"If his arguments are shaky, he will step down from the pedestal on his own when he realises that this decision is inefficient and unnecessary." - The sad part about this person is that when presented with this type of feedback, they always double and triple down despite the facts on the board, it is really odd and risky behavior to be honest.

At the end of the day, I am the person who has to put him in his place, so I want to make sure I am not missing a major point of consideration with celery.

[Tinos]

Oh boy this sounds like quite the situation. You obviously understand the internal affairs far greater than I, but you did say "I am the person who has to put him in his place". Not saying this is the solution but if you're the person who put him in, you're probably the best person to kick him out?

Again, I don't want to meddle with decisions involving total stangers' careers but philosophy that the CEO at my old company had (which was a startup optimising for 100x efficiency) was "hire fast; fire fast" and his startup is doing pretty well under that mantra!

[sigotirandolas]

I can't claim to be able to say much, but if it's anything like my experience, it can be an organizational problem.

I've seen this happen where there is something like an "architect / senior" vs "normal programmer" role split, where the architect can sell some stupid idea that sounds good, or make a proof of concept implementation, then throw it over the wall to someone else who needs to maintain and fix the mess.

Or if practices like code review or testing aren't done or aren't done properly, which is a way to veto or slow down bad ideas.

If the problem "trickles down" from the top of the company and you have no leverage there, I don't think there's any easy way to fix it, if any.

[theorlandog]

That's definitely the core point. To this guys credit, he figured out that in the 2020's you don't need to be technically correct to sell solutions; you just need to liter your documentation with emojis and horse shit to win people over.

I just want to make sure in this case, I'm not missing out on some super special sauce or advantage that celery provides. Consuming a queue seems trivial that I am worried I am missing something.

[sigotirandolas]

(The usual disclaimer: Take what I say with a grain of salt, I am just speaking from my own N=1 experience here and I am not any good at management).

To me the discussion about Celery vs polling the queue didn't seem like the core theme, in that they are different tools with different trade-offs and you most likely can end up solving the problem with any of them even if you pick the wrong one (and learn from it!). Rather the problem seems to be that you think that your team is picking the wrong tool and you have no power to change it, and you feel that there isn't going to be any consequence, change, reflection or learning whatsoever if/when that's shown to be the case because you have been there before.

At least for me this was a pretty big component in burning out in the past, I put a lot of care and effort into what I do, so when I felt that someone else was continuously and stubbornly making bad decisions and I had no mechanism/leverage to push back against that, it caused a lot of resentment and burned me out.

[ano-ther]

Sorry, I cannot add much on Celery.

If "every time one of his ideas is implemented it turns into a flaming bag of complexity dog shit, and everyone else is left holding the bag to clean up," then it might be better to discuss how you make and stick to decisions in the team, rather than following your fellow engineer down his fad-of-the-day.