[lightcatcher]

I was an intern at Mixpanel at the time (but not the author of this article or involved in the discussed rewrite). For context, the company was then <10 people. (I'm surprised to see anything from my first professional software experience on top of HN!)

My recollection of events (of 8 years ago): The Erlang endpoint was written a year or two earlier as someone's first Erlang project, in a tiny org with no Erlang experience. The endpoint worked well enough and people moved onto other projects in the mostly Python codebase. Eventually, it became painful to debug or add features to this endpoint because no one was particularly proficient in Erlang. Ankur (author of the post, intern) then rewrote it in Python.

I wouldn't read this as a negative about Erlang or a positive article about Python. I'd instead read it as "use a language that makes sense for your organization and already existing codebase".

[erlangNewb]

Seems like an oversight that nobody thought to get proficient with erlang. As someone who has started learning it, this seems like the obvious choice.

[diminoten]

From the company's perspective, you'd need at least 2 people doing that, preferably everybody. Bus factor of 1 isn't great, and when the rest of the team already knows Python, the benefits of switching to a language everyone knows is huge.

You're also presupposing Python is inferior to Erlang, which I'm not really going to argue about here, but I think we can all agree that Python isn't so bad that the decision to continue to use Python substantially and negatively impacted the org's ability to succeed.

[macspoofing]

>Seems like an oversight that nobody thought to get proficient with erlang.

If it's a component that nobody owns, and nobody cares about, then I'm not very surprised.

>As someone who has started learning it, this seems like the obvious choice.

The other obvious choice is to rewrite it.

[bonif]

Thanks for this comment, gives context and explains "why we switched from X to Y", answer: I did not know X, or it didn't make sense at the time. It does not mean X is bad, or Y > X (as usually suggested in this kind of articles)

[bradleyjg]

In general language diversity at a company is a big negative. Obviously sometimes there are even bigger negatives that mandate overriding that rule--you aren't going to use Java on the front end or javascript on the backend, that would be madness--but you never want two languages in the same niche. That's a cost with no benefit.

If you have some developer that wants to write something in a new (for your company) language, or even worse goes off and does so without asking, because he wants to try it out, you probably want to fire that guy. Either he doesn't understand the downsides or he does, but doesn't care.

[cellularmitosis]

Some dev: "Hey, I noticed one of our more complex API endpoints had some pretty bad response latency, so I dug into it a bit. It turns out our PHP/Symfony/Doctrine stack was generating over a hundred SQL queries, all of which were being performed sequentially because we don't have a good per-request concurrency story in this tech stack. So I wrote an alternate implementation in Clojure which has a great concurrency story and cut the latency by 75%."

lol yeah that guy should totally be fired.

[delian66]

Another developer: "...our stack was generating over a hundred SQL queries, all of which were being performed sequentially because we don't have a good per-request concurrency story in this tech stack. I just bypassed the ORM and wrote a slightly more complex SQL query, so that there is no more N+1 query problem in the backend. The latency of this end point was cut by several orders of magnitude, and our database is not overloaded by parallel inefficient queries, run by clients with 'great concurrency stories'."

[Izkata]

Or, "here's a completely supported not-well-enough-known ORM feature that does that, that I learned about when it looked like I wasn't doing any work".

[cellularmitosis]

From a technical perspective, bypassing the ORM is a great solution (and if you can get the number of queries down to one, there’s no longer any need for concurrency).

Unfortunately, politically, the kind of shop which fires a dev for choosing another platform is unlikely to approve a PR which bypasses the ORM (maybe also a fireable offense!)

[theamk]

If there are no other people who can maintain that stack, maybe (firing is too extreme, I’d just ban the Clojure code in the production)

Writing initial version of the code is only a small part of it. Maintaining and enhancing it is much harder.

What is the company going to do if the new endpoint went down and the author is on vacation?

[cellularmitosis]

> just ban the Clojure code in the production

A dev ran up against the limitations of one platform and solved the problem by choosing a platform which didn't have those limitations. And management responded by banning the new platform.

How long do you think that dev is going to stick around in an environment like that?

Ultimately this comes down to what sort of environment you want to foster. One approach leads to an environment where devs have lots of freedom, and that creates certain problems. The other approach eventually leads to an environment where even the choice of editor / IDE is dictated, and that creates a different set of problems.

I'd favor attracting a bunch of smart folks and give them as much freedom as possible. I'd much rather deal with the kind of problems which that sort of environment creates.

[theamk]

I am not sure why you think it is "management responded" -- it is all about other people on the team.

Right now, I am responsible for maintaining one of our internal services. If someone tries to rewrite part of it in Clojure without extensive internal discussion first, I'll do all I can to stop it (and I am not "management" at all). And most people on the team will agree with me.

If the original dev would not want to stick around after that, good for them. If they do not care about teammates at all, they would likely be happier working alone anyway.

After all, single-handedly moving to a new technology and then forcing everyone else on the team to support it is impolite and annoying. Clojure is nice, but I don't want to be forced to learn it because the server went down on my shift.

(this is different if the service does not need to be up all the time, or the company is very small. But if you are talking about API endpoint being overloaded, the chances are, neither of this is true)

[cellularmitosis]

“If someone tries to rewrite part of it in Clojure without extensive internal discussion first”

“single-handedly moving to a new technology and then forcing everyone else on the team to support it is impolite and annoying”

The dev you’re describing has indeed made some bad choices and I wouldn’t want to work with that guy either.

What I had in mind was more along the lines of “check out this thing I made, what do you think?”

Totally ok responses would include: “let’s just bypass the ORM here instead”, “let’s divert 5% of traffic to it and see what happens”, “this stays a prototype until you reach a bus factor of 2”, etc.

[Silhouette]

A lot of these objections depend on context, though. The culture at large organisations taking on large and/or long-running projects is necessarily different to the culture at small organisations that can't afford the same overheads and wouldn't enjoy the advantages of greater redundancy or interchangeability anyway.

[andy_ppp]

The problem here is not understanding your database not about making a system in an unsupported and probably undeployable thing to prove how clever you are...

[cellularmitosis]

> "to prove how clever you are"

I see this a lot and I just don't get it. Is a mechanic who uses air tools trying to "prove how clever he is"?

Knowing more than one tool allows you to choose the right tool for the job. Assuming arrogance and a motive of "proving cleverness" sounds like a great way to pick a fight.

[verinus]

this is not a valid comparison.

a mechanics' tools would be IDE, build tools etc- I'm not interested how every dev uses personal level tools.

But just writing a program with a new dev stack without thinking of the possible consequences is just plain unprofessional.

[theamk]

Air tools are fine -- after the car left mechanic's garage, I cannot tell if air, electric or manual tools are used.

But if I give my car to mechanic for carb cleaning and tune-up, and get it back with fuel injection system and explanation "it is better anyway", I would be very unhappy.

[UweSchmidt]

There are no parallels in the real world to how programmers chose and use languages, frameworks and tools.

[TelmoMenezes]

He should get fired for saying "concurrency story".

[_pmf_]

> lol yeah that guy should totally be fired.

Yeah, introducing a new ecosystem to mask bad DB design or bad queries because she is lacking basic SQL and relational foundations is how I envision the typical 10x programmer.

[collyw]

I hope you are being sarcastic.

[e12e]

Maybe not. But if the alternative was spend one tenth of the time to cut the redundant sql queries down a bit and cut latency 50% - leaving a codebase that anyone on the team could further develop and debug... Then "that guy" might stil be in the wrong...

[collyw]

Rather than making thousands of requests in parallel, wouldn't reducing the number of queries be the more obvious choice.

[cellularmitosis]

Both would be ideal.

[nojvek]

Language diversity is a pragmatic balance. Sometimes a language offers superior abstraction or some guarantees that eliminate a whole host of bugs and manual effort.

It’s always worth trying a hypothesis in a small spike and reliably proving whether it will scale to the rest of the org and the impact will be worth the effort.

But you’re right, there should be one official supported way of doing some kind of thing at a company.

[lightcatcher]

I think of it as a trade-off between the tool perfectly designed for the job and the tool that can be used for the job of which I'm already an expert user.

As an example, my current team's codebase is Python and C++. When we need to do some basic Linux scripting (check if this file exists, if not send an error email), the two prime candidates are Bash and Python. Bash might be exactly designed for this type of thing, but a lot of the team would need to Google stuff like "bash logical and of two booleans" for Bash where they already know the Python syntax. My current rule of thumb here is "<10 lines of code: use Bash. else use Python". More generally, I often try to err on the side of using the tool I know than the unknown tool that might be perfect for the problem.

[Silhouette]

If you have some developer that wants to write something in a new (for your company) language, or even worse goes off and does so without asking, because he wants to try it out, you probably want to fire that guy. Either he doesn't understand the downsides or he does, but doesn't care.

What if it has the potential to be a much better tool for the job than the mainstream options, and your developer thinks it's worth trying it to find out?

I've seen this situation with adopting a relatively unusual language for a task that needed particularly high reliability. Yes, it was a concern that only one person currently working in the organisation knew much about it and that it might be harder to hire help if more developers were required later. On the other hand, that software was developed faster than the similar system it was due to supersede, and it had a much lower rate of reported bugs once deployed.

Of course you don't want to hire people who intend to pad their resumes with extra buzzwords on your dollar when it has no business benefit, but equally, if you never use tools that aren't chosen from among a small set of mainstream options, you'll never outperform what you can do with mainstream tools.

[pjmlp]

Software development is much more than just programming languages, I have seen teams outperform with lousy languages thanks to a great team spirit and understanding of customer needs.

[Silhouette]

For sure. A good craftsman will do good work with any reasonable set of tools.

They'll still achieve their best possible results and performance with high quality tools of their own choosing, though.

[bradleyjg]

I guess my answer would be that this is the kind of decision that requires being able to see the big picture. Is an individual developer in a position to weigh future recruitment challenges?

My remark about firing a person merely wanting to write something in a different language was off base and end up derailing the discussion. But the point I should have made, and stick by, is that this decision is properly made at either the company wide level, or for large companies at the level several rungs up the ladder. It isn't at all, like someone suggested above, equivalent to picking a personal IDE.

[Silhouette]

Is an individual developer in a position to weigh future recruitment challenges?

Maybe, particularly if it's a smaller organisation.

In any case, it's obviously a management decision at some point, as all things are. However, it seems quite reasonable for me for a developer to advocate using a more suitable tool even if it's not an existing choice/standard within the organisation, and if they make a good case, it seems quite reasonable to me that management might then back them despite the downsides if the upsides appear greater.

[tomjakubowski]

You could fire them or you could just help them understand something that they don't. Sheesh.

[bradleyjg]

Fair. That part was over the top. But I do think employees pushing their own intellectual passions or resume building pursuits over the interests of their employer are pernicious.

If it’s out of ignorance, then sure by all means educate.

[collyw]

It's a two way street. Employers always want the most up to date trendy frameworks on their employees CV's. I imagine someone that has been working on a legacy VB6 app for the last few years will struggle to get his CV looked at.

[detaro]

> If you have some developer that wants to write something in a new (for your company) language, [...] you probably want to fire that guy.

Now that's

> a cost with no benefit.

[michaelterryio]

Erlang excels at being used in mixed-language shops. Erlang is an incredible control plane language.

[jacquesm]

> or javascript on the backend, that would be madness

It seems madness is pretty common.

[k__]

True.

For most projects, it's good enough.

Also, I worked for many shops that had a two language policy, one for development speed and one for performance.

For example, PHP and C or something.

They had many PHP devs and a few C devs and would write anything new in PHP and then later replace the few performance critical parts with C implementations.

I thought about doing this with JavaScript and Rust in my projects, but somehow I never got to the point that JavaScript became the bottle-neck.

[Daishiman]

It is. Mistakes take a far longer time to be understood than fashion.

[teilo]

My company is an example of this, and in a way that is quite different from that laid out in the article. We actually began with a Python implementation for our integration / API server. But this was well before asyncio and eventlet and kestral. It didn't scale well, and we quickly outgrew it.

We implemented its replacement in Elixir, a solution we are still running. It is written by a developer we still have, is well organized, follows proper functional paradigms, and fully leverages OTP/Genserver. It is resilient, and so far as the one two developers we have working in it at present, easy to maintain. We love it.

But we also are switching back to Python. The reason is simple: We only have two Elixir developers (really only 1.5), and we have far more work than they can complete. And when something breaks (usually because something changed in the various systems being integrated), we have one guy who can fix it, and get our production queues flowing again.

We have looked for more Elixir developers. They are rare beasts. But we have a lot of Python developers. And now that we have asyncio/eventlet/etc., the landscape has drastically changed. In Python, the various teams can create their own worker modules, and do so more efficiently because they know their own workloads far better.

Language diversity IS a problem when it impedes development and puts production at risk.

[bcbrown]

> We have looked for more Elixir developers. They are rare beasts. But we have a lot of Python developers.

Was there any consideration of training up the Python developers on Elixir?

[teilo]

That's the 0.5 Elixir developer. Let's face it: It's easy to become a poor Elixir developer, but those capable of becoming a good Elixir developer such that it's worth the cost of effort to train them, are uncommon.

As the CIO, where the buck stops, I learned Elixir well enough to write my own Genserver-like implementation from scratch, and thoroughly enjoyed it. I wanted to be able to understand the code base. But I am still a far better Python developer than I will ever be an Elixir developer. It's all a question of time utilization.

Due to the modular nature of our integration, it's relatively easy to move from one platform to the other, one module-at-a-time. And in the end, the cost of getting a new Python-based platform up-and-running far outweighed the cost of trying to retrain Python devs.

[scarface74]

We have three approved languages - C#, JavaScript and Python.

Of course C# you get all of the advantages of statically typed languages, so for major programs where you wil have multiple developers, it’s the go to language. But for smaller scripts like event based lambdas and scripting type scenarios, C# is overkill. Besides, each piece of AWS functionality is in its own NuGet package as opposed to one package for everything for JavaScript (Node) and Python.

JavaScript - our product is web based. Every developer is expected to know JS. So if someone decides to do a simple Microservice or lambda in C# or JS, it’s okay. Whichever they feel like doing, it’s okay.

We have another team that does a lot of ETL type processing. I doubt that many people would argue that JS is a better language for it than Python. If you are on that team and need to write something, you can choose whichever of the three languages you wish.

[nathan_long]

> In general language diversity at a company is a big negative

I agree. It really increases the difficulty of understanding the whole system. But it's a negative that has to be considered alongside potential benefits.

> If you have some developer that wants to write something in a new (for your company) language, or even worse goes off and does so without asking, because he wants to try it out, you probably want to fire that guy.

I get what you're saying; resume-driven development can ruin your systems. But at some point between "we wrote this new banking system in COBOL" and "every developer who knows COBOL and could maintain our old system is now dead", there needs to be an accepted way to experiment.

[aiisjustanif]

Depends. If it's a internal, innersourced, tool for everyone. Have at it, use another language. If it's my critical application, explain why we use only one language for it and don't fire them.

We teach, not react.

[ummonk]

Why would it be madness to us javascript on the backend?

[collyw]

Because there are far better languages to choose from.

[oconnor663]

I think Facebook chat had a very similar experience. The first version was written by someone who liked Erlang. Later on, when some performance issues came up (probably related to GC), no one knew Erlang, and all the high performance backend code in the company was in C++. So it made sense to rewrite the thing.

[trab]

Then they acquired whatsapp that was written in Erlang.

[ludwigvan]

And I think I had read that Whatsapp used Erlang because Facebook Chat was using Erlang and they thought Facebook knew what they were doing and decided to use the same tech. What a twist

[reducesuffering]

Where did you read that? I believe Whatsapp used erlang because they leveraged ejabberd [0] for XMPP. Wouldn't surprise me if this is why FB originally used Erlang for the chat servers too. [0] https://en.wikipedia.org/wiki/Ejabberd

[darkr]

Yep. Facebook chat was originally based on ejabberd.

Source: https://web.archive.org/web/20121224094022/http://cufp.galoi...

[fineoldcannibal]

I believe it was because of the Beluga acquisition which was in Erlang

[mst]

Your recollection is valuable, and thank you - but contra a large chunk of the rest of this thread, the article was very clear to me that it was "our erlang was crap erlang and we didn't know how to make it not crap so we switched" which is totally reasonable and didn't seem like a negative towards erlang at all. I've supported rewrites from a language I liked more to a language I liked less for similar reasons, and what Ankur did seems like an entirely sane decision.

But even if I didn't need the clarification, the additional data was awesome, so my thank you remain sincere :)

[rlander]

So, in a Python-only org, someone with no Erlang experience decided to ship his first ever server to production, thinking Erlang would be a magic bullet to “scalability issues”. Obviously, there were maintainability issues and the org, blaming the language, decided to switch back to python.

To me, the fact that a novice was able to deploy production code that went on unmaintained for months, running a crucial part of the operation, speaks positively about Erlang.

[robocat]

"the org, blaming the language".

I didn't see anywhere in the article blaming the language - can you find a quote to support your assertion?

From the article: "After two years of iteration, the code has become difficult to maintain. No one on our team is an Erlang expert, and we have had trouble debugging downtime and performance problems. So, we decided to rewrite it in Python, the de-facto language at Mixpanel."

[_RPM]

Is it common for an intern to get such an impactful project?

[exelius]

Yeah; it’s actually pretty easy work. Interns have a lot of uncommitted time, and they’re going to leave soon so you don’t want to integrate them into the team too much. It’s also way easier to build to a well-defined spec (which an in-production endpoint would likely have).

So you let them rewrite something that already works. If what they build is good, you‘ve just knocked out a good chunk of technical debt and found someone you probably want to hire. If not, they don’t get an offer. Either way you probably didn’t pay them much to begin with so there’s a lot of upside and not much downside.

[ajross]

The question isn't whether or not the work is easy for the intern, it's whether or not it's appropriate. And this cuts both ways: a company that is doing serious core refactoring/redevelopment on the back of an underpaid "intern" is probably exploiting that labor in an unfair way. If it's in fact an unpaid internship, then they're straight up in violation of the Fair Labor Standards Act.

And, of course, a company that puts this kind of work on the back of a temporary worker is not treating its own codebase with the respect it deserves. This is rolling the dice with your expertise store -- you have to hope they did it well, because they aren't going to be around to answer questions if they didn't.

[kbutler]

If they didn't build it well, you throw it away, and you're only out what you paid the temporary worker and your code review/acceptance test.

The point is this was a well-specified, independent component with an existing functional implementation. This makes an ideal intern/temp/first project, because of those factors - it doesn't require deep knowledge of the organization or other services in the environment or business requirements. You build this, and if we like it, we use it and may give you a job offer.

Unless given other information, you can assume the company complied with the law with respect to paying the intern and the intern accepted the internship offer, and it sounds like he got a great experience out of it.

[ajross]

> If they didn't build it well, you throw it away, and you're only out what you paid the temporary worker and your code review/acceptance test.

Yeah, and that logic is almost inherently a violation of labor law. The whole idea behind allowing "internships" at all is that the intern is deriving value (education) from the relationship that isn't captured by wages alone. The test for whether it's legal involves how much they are supervised by the people who are supposed to be teaching them.

Handing out throwaway projects like you posit is an easy trip to a class action suit.

[Stratoscope]

I think you may be mistaken about this. If it helps any, I wasn't one of your downvoters; I upvote comments that may be incorrect but lead to an interesting and informative discussion.

I live with an HR exec and read some of this thread to her while she was cutting rhubarb for something she's baking.

She said (paraphrased), "Are you kidding? This is a perfect project for a summer intern. They won't have to take three months to get up to speed with all your internal systems, and of course there is educational value - they will get to see how some talented developers tackled the problem in one language while they translate it to another."

[lightcatcher]

> a company that is doing serious core refactoring/redevelopment on the back of an underpaid "intern" is probably exploiting that labor in an unfair way

> Handing out throwaway projects like you posit is an easy trip to a class action suit.

So if an intern shouldn't do work on the core product but also shouldn't do throwaway projects, what do you suggest an intern should do? What do you consider an "underpaid" intern? Would you consider 75% salary of campus new hire as underpaid?

[exelius]

Most real internships pay under market rate, but not insanely low. A few years back, going rate for an undergrad CS internship was ~$25/hr or so.

If you’re recruiting on campus, you’re competing with other companies for the most promising candidates. If you come in with an unpaid internship, you’re going to get the leftovers.

[bradleyjg]

Even ten years ago, I doubt it was an unpaid internship. They weren’t common in tech going all the way back to dotcom. And if it was a paid internship, labor law doesn’t care. As far as it’s concerned that’s a job like any other.

[robocat]

GP didn't suggest "Handing out throwaway projects", you did: I think you misparsed their comment.

You ask the intern to do work that you want them to succeed at and you think they will succeed at: like the example given.

[detaro]

Is your point of contention the "your code review/acceptance test." as the only time investment?

[scarface74]

As far as I know, unpaid internships have not really been a thing in software development.

I was an intern back in 1995 and they paid us $10/hour ($16.88 inflation adjusted) and they provided housing on a college dorm. To put that in perspective, the $650 I made after taxes every two weeks was the cost of tuition every quarter at the state school I attended. I made enough that summer to cover my entire senior year including books.

[znpy]

No idea, but that seems like a reasonable choice in that case.

Yes the project was impactful, but there already was a reference implementation with a reference performance target. Worst case scenario, they would have been back to using the old implementation.

[macspoofing]

Depends on the intern, and the component. If the intern is talented, the component is well-tested and the mentor or supervisor can do an effective code-review - then why not?

[notdang]

lightcatcher did not say that the work was done by an intern

[hnra]

> Ankur (author of the post, intern) then rewrote it in Python.