[wpeterson]

This article is dangerous in romanticizing the “not invented here” culture at many big tech companies and seems rooted more in the 90s than present day.

The world of open source tooling and easily re-usable SAAS offerings means everyone has access to the best tools, whether you’re a small startup or a big company.

Anyone who longs for internal, corporate tooling baffles me when they can use things that actually have polish, user experience and likely better implementations under the hood.

Companies should spend their time/energy building things unique to their problem domain, not weak also-ran corporate tooling.

[barrkel]

The mistake you're making is thinking that the public tools are the best tools.

There are reasons that make them the best for many, if not most, companies: more investment, more mindshare, easier to hire employees with prior experience, and so on.

But there are also costs in having a wobbly stack of glued together stuff, especially if the parts aren't quite right for your goal.

Sometimes the best tool is more focused, more vertically integrated, in a different language or for a different operating system because those choices integrate better with the rest of your stuff.

The constraints of the company are also part of the problem domain. Using the wrong tools can be quite expensive, and the public tools may all be just a bit wrong in a way that compounds.

[kelnos]

Where I work, the majority of our tools were built in-house, mainly because we started (2008) before there were good open source or even paid options for most of it all. As good options started to appear, we found that we couldn't adopt them, because there was a mismatch in concepts/fundamentals between what we'd built and what was out there. We've evaluated a lot of things, and for many of them, we end up realizing that integrating them with our systems would require a hard fork, and so we'd lose most of the benefit of using it.

Frankly, it sucks. Our tools are mostly very good, but it took us a long time to get there, and the internal fights over getting funding to really invest in our internal platform have been exhausting for all involved. I get that it's not zero work or zero time to use something off the shelf, but as someone who has been playing in the grass on the other side of that particular fence, it takes a lot of work to keep that grass green.

[blandflakes]

The internal build tools I had access to at Amazon are unrivaled in the public domain. I DO miss them, at every job.

[webmaven]

I'm asking this sincerely: What is/was so great about them?

[blandflakes]

It was probably some combination of how well-integrated and how opinionated they were.

The build tool was multi-language. It allowed depending on arbitrary packages which had been imported to the Amazon package repository. It allowed package owners to annotate the packages with guidance - experimental, deprecated, forbidden (in the case of security issues), etc. You could also declare conflicts, which would notify consumers at compiletime and force them to resolve in some way. The tool deferred to a number of standard build tools in whatever language(s) you were using; it was just about getting and packaging the dependencies.

When you committed code, a build was submitted to a distributed build system. It would run your build, and then it would run the builds of every package which declared a dependency on your version. If those builds failed, your build failed (so, bump your version or make your change backwards compatible). On completion, it imported an immutable bundle of your artifact + dependencies to the deployment infrastructure.

This is the part I miss - the build stuff was great but I largely find that open source and paid options aren't so bad here. What I miss is how easy it was to manage the journey of a built package to your machines. They had a tool called Pipelines that had a visualization of this progression. Each stage, with associated machines, was linked here. The tool knew how to add stages of environments, each with their own set of deployment configurations. You could set up approval workflows: integration tests, manual approvals, etc. You could feed one pipeline into another. For each artifact, you could configure autobuilds into your pipeline, so that new versions flowed as long as tests/approvals allowed. There was support for sanity checks: if those failed, the tool would automatically rollback. In fact, if any stage fails, your pipeline would block, and you'd be notified. In some cases, a newer build that was fully functional could unblock your pipeline.

Pipelines was a pleasure to use - it really just got the hell out of my way, and nothing I've used since is as simple to integrate with.

And tying this all together, there was a tool that would allow you to initialize the end to end infrastructural pieces needed. You go to a wizard, tell it your language and purpose (webapp, CLI tool, service) and it initializes a repository, package, and pipeline with environments. Get a coffee, come back, and Do Your Job.

I imagine similar things exist at places like Google, but man everywhere else I've worked, so much developer productivity has been lost not even approximating the level of "let me do the interesting stuff" that Amazon provided.

[webmaven]

Awesome, thanks for the informative reply!

[walterbell]

What's the closest OSS equivalent and how could they be improved?

[blandflakes]

Just responded to another comment - for the part I miss most, Pipelines, Spinnaker is probably closest (I'm aware of Concourse, etc as well). But Spinnaker is horrific to stand up yourself (especially at my company of 8 engineers), and it's actually too generic. The primary advantage I believe is that the tools were exactly built to work well at Amazon, with Amazon's infrastructure.

[drodio]

Armory (W17) can help with spinnaker. We offer a managed version for exactly this reason. Probably still way too heavy for an eight person engineering org but helpful for larger companies.

[goatinaboat]

Anyone who longs for internal, corporate tooling baffles me

A previous employer had an all-proprietary stack: language, IDE, version control, deployment system, database, job scheduler. It was, frankly, amazing. The most productive environment I’ve ever worked in. Time-to-market was this company’s competitive advantage and this stack let them leave their competitors in the dust, scrabbling for a distant second place.

everyone has access to the best tools,

It’s 5 years since I left there and the FOSS tooling we use at my present company doesn’t come close, and it never will, because choices are driven from the basic assumption that FOSS is always best, and it just isn’t true. We are at least 5 years behind where the previous company was when I left it and they won’t have stood still in that time.

[cm2012]

As a marketer, I always look for no code solutions first. API integrations? Use Zapier. Email form integration? Sumo or similar. Triggers? Google tag manager. Etc. The reason being that the dev team at any company never has time for new projects.