[danpalmer]

> I don't know any more than this paragraph or two explains, but it sounds like NIH syndrome. If you chose to write your own solution just because a 3rd party one was complicated or expensive, you've underestimated the complexity and expense of developing and supporting new software. Not only do you have software developers developing your business products, but now you have software developers developing the IT tools that support the software developers writing the business products.

I think the part "Leveraging in-house primitives" is the important bit here. It sounds like they have an in-house task queue, so introducing another external one like Celery might add a lot of operational complexity rather than piggy backing on existing infrastructure. As for the reason for the existing infrastructure not being Celery, I suspect at Dropbox's scale tools like Celery start to break down.

> Or you could, like, install Jenkins, write your tests, and do all this without writing your own distributed job queue system.

They likely haven't written their own job queue, they've just used an existing one. Again, the operational overhead of Jenkins is unfortunately very non trivial.

[swish_bob]

> Again, the operational overhead of Jenkins is unfortunately very non trivial.

Not to mention the overhead of running Jenkins at the kind of scale they're talking about here. We're not talking about small numbers here.

[peterwwillis]

I don't know other people's experience with Jenkins, but I didn't find it to be much overhead. It's got all the capabilities you need to run at high scale, assuming you use it properly, like using the DSL and containers, and managing the configuration and deployment as code.

Jenkins X, for example, is inherently cloud-based and dynamically scaling. If you need a generic job queue to grow with the organization, that's what it does. And you can even pay someone to set you up, reducing lead time.

But there's also a dozen similar systems out there, so you're not even limited to this one. There's lots of choices and really no need to make a new one.

Re-using an existing component to make a new tool is still an independent software development project, which will almost never be as cheap as integrating a finished tool.