[hitekker]

I imagine most people upvoted because it sounded smart.

It's fascinating how easily cruelty can be popularized by using the right, nice-sounding words. Coating/mask your bile in a rhetoric popular with a community, indirectly imply some terrible things, perhaps obfuscate anything that could raise any uncomfortable, thoughtful questions, and presto! You'll have the right set-up to manufacture consensus.

[tptacek]

It's funny because what got me is how dumb it sounded: the words "such a colossal error as writing their own language" made me think "that's the kind of thing a mid-level IT manager at an insurance company would say".

[Alupis]

It is usually a "colossal error" to write your own in-house closed-source/proprietary language, no matter how small or large the language is.

The main reason is exactly as the article states, maintainability.

> As time wore on, our technical debt finally began to come due. Compilers like Wasabi and their associated runtime libraries are highly complex pieces of software. We hadn’t open-sourced it, so this meant any investment had to be done by us at the expense of our main revenue-generating products. While we were busy working on exciting new things, Wasabi stagnated. It was a huge dependency that required a full-time developer — not cheap for a company of our size. It occasionally barfed on a piece of code that was completely reasonable to humans. It was slow to compile. Visual Studio wasn’t able to easily edit or attach a debugger to FogBugz. Just documenting it was a chore

[tptacek]

What am I missing? This is an internal language designed as an incremental improvement over VB that gave them cross-platform common codebase. It lasted 10 years: that's 19,932 in SaaS years. When they transitioned off of it, they did it not with a rewrite, but with mechanical translation.

It seems like a spectacular success story.

[hiharryhere]

Colossal error vs spectacular success story. Yay hyperbole.

I'd say the actual experience is somewhere in between. Sure it enabled them to support client requests to be cross platform and proved useful for a very long time, but what was the broader opportunity cost? Did supporting this proprietary infrastructure eat up resources and prevent them from exploring other ideas? Probably.

[tptacek]

Big projects are routinely ported once or twice to new languages. Normally, moving a project from one language to another entails a full rewrite. Because the process of building their own language had the side effect of instrumenting all their code (the transpiler was custom-designed for their application), they were able to write a program to do the port for them. That's a win I don't see captured in the original blog post, and that's all I'm pointing out.

[krallja]

I don't think it ate up any more resources than, say, MySQL support. Fog Creek employees have produced tons of innovation since Wasabi was introduced, including brand-new companies like Trello (2011/2014) and Stack Overflow (2008), as well as in-house products like Kiln (2008), WebPutty (2011), and Make Better Software: The Training Series (2009-ish). None of these projects were particularly resource-constrained by having to do some compiler and runtime maintenance in the process of building FogBugz.

[muench]

Thank you for the most insightful comment on this story. I find it bizarre that so many are unable to understand that making the decision to 'kill' wasabi today does not necessarily mean that it was a mistake.

This whole story is a fabulous insight into software development for business over the long-term.

[to3m]

Perhaps even a case study of technical debt done right?

[duncan_bayne]

Sure, but success stories only make the people who worked on them look smart, whereas post mortems give everyone a chance.

[bm1362]

I mean, adding an esoteric tool into your development process -- one where an employee has to write a book about it's quirks -- sounds like a failure for the guys in the trenches.

[tptacek]

On practically every software project in the history of software projects that have lasted more than 3 release cycles, there is a person on the team who could write a _${Some Component}, The ??? Parts_ book. Nobody ever thinks to do that, because _${Our Report Generation Library}, The ??? Parts_ is super boring.

The reality though is that line-for-line, a transpiler is probably not much harder to write than a serious report generation tool. I agree with the commenter upthread, who thinks this is a result of people simply never having tried to write a compiler before.

[Alupis]

> It seems like a spectacular success story.

It might be, but for what many would argue are the wrong reasons, maybe even unnecessary.

> This is an internal language designed as an incremental improvement over VB that gave them cross-platform common codebase.

The problem is that at the time, there was already several cross-platform technologies in existence, many of which were being developed in the open. Utilizing one of these technologies would have allowed FogCreek to focus on what they do best, make software. Instead, they took a proprietary uni-platform language and attempted all on their own to make it cross-platform capable - which led to years of maintainability issues.

> It lasted 10 years

They gained an early advantage of not having to throw out the codebase and start over, yet they bought themselves 10 years of technical debt which continued to pose a burden on the small company. Many would argue that biting the bullet early on and switching to an open, community-driven cross-platform language/environment would have yielded much more return on the initial investment.

> When they transitioned off of it, they did it not with a rewrite, but with mechanical translation

Yes, that is an achievement, but again, for the wrong reasons.

[tptacek]

I feel like you could take this comment, make very minimal tweaks, and deploy it in any language-war debate. "They succeeded with Golang, but for all the wrong reasons. They gained an early advantage but bought themselves 10 years of technical debt that a more modern cross-platform language would have spared them".

And I feel like when you get to the point where the best arguments you can make against something are isomorphic to the arguments you'd make against mainstream languages in language-war debates, that's a win condition.

[Alupis]

Sure, FogCreek thought it was a good idea at the time, but over the years it became a significant burden, to the point they had dedicated staff working on just keeping Wasabi alive. Time was even spent writing an internally distributed book about the caveats of the language!

I know you will dismiss this as "routine", but it's not...

For a small company, this is an enormous waste of time, money, and energy.

A big company like Google or Microsoft can afford to throw developers by the dozen at internal proprietary languages and not even blink -- but according to the article, FogCreek did blink every time they had to dedicate time to fixing it. It took time, money, and energy away from their core business - making software.

That's a lose condition.

FogCreek should have bit the bullet and re-wrote their application in an open, standardized cross-platform system. They would have been able to spend zero time worrying about the language, and 100% of their time worrying about their application. They could hire engineers off the street and have them produce in days-to-weeks instead of weeks-to-months. They would have saved themselves an enormous amount of time, money, and energy invested in a language that is now dead anyway.

It may have seemed like a good choice back when the decision was made, but in hindsight it appears to have been a very poor, short-sighted choice.

[StavrosK]

I think his point can be summarized as "it's better to have to maintain your software than to have to maintain your software and the compiler for it", which is hard to argue against.

Then again, given that they had the codebase already, writing their own transpiler sounds like it was the best option at the time.

[cwyers]

> The problem is that at the time, there was already several cross-platform technologies in existence, many of which were being developed in the open.

FogBugz wasn't written in any of those languages though. And it isn't like Spolsky's "never rewrite from scratch" views are unknown here.

[ghshephard]

"It is usually a "colossal error" to write your own in-house closed-source/proprietary language"

I'm curious if anybody on this thread who has written more than three or four compilers/parsers would agree with you.

Depending on the task, the only solution to some problems is to write a custom/proprietary language (whether it's closed source, of course, is up to the company).

[chucksmash]

> Depending on the task, the only solution to some problems is to write a custom/proprietary language

Is a bug tracker one of those problems?

[slantyyz]

>> Is a bug tracker one of those problems?

But "bug tracker" is not the problem that was being solved.

The problem was taking a big pile of legacy code and translating it to more than one platform vs rewriting the entire app from scratch in a new language that was cross platform.

It just happened that the legacy code was for a bug tracker, but it could have been for anything.

[wglb]

It is usually a "colossal error" to write your own in-house closed-source/proprietary language, no matter how small or large the language is.

Really?

I don't think so, having done this once, to the great success of the company. They also wrote their own database. The compiler was maintained by a team long after I left the company.

Software is hard. There are more interesting, and hard, problems than pushing the value of a field from one subsystem to the other.

[duncan_bayne]

Whereas, complex systems built on popular OO frameworks never have issues with maintainability.

/me deactivates snark mode

I see the issue exactly the other way around.

If you can build a domain specific language that lets you express concepts in a clear way free of boilerplate or conceptual hackery (ORMs, for example) you will wind up with a much lighter maintenance load than the equivalent functionality built on an off the shelf framework.

Of course, there's nothing stopping you from using the two as appropriate. Simple CRUD app? Rails. Need to express a very complex domain in a readable, easily maintained form? Custom language time.

[arielby]

Most things that have lots of reverse-dependencies require a significant amount of maintenance. Compilers are not much different from "common" libraries, or special-purpose frameworks, in that respect. Also, writing a direct-to-assembly compiler is probably not a wise idea.

[kamaal]

I'm not saying you are wrong. Building the era of custom tools just for your company, only to do specific jobs is long gone. This is for a lot of factors of risk in a business.

It would work, if management didn't treat programmers as a replaceable cogs in a wheel. But the day you seek to make the craft of programming a commodity that could be done anyone, you need a ecosystem whose knowledge is available to everyone. Only then would you get reasonable expertise at affordable prices to finish your projects.

The opposite is to make so programmers so special that the knowledge of specific tools is available only to them. This definitely puts programmers in a lot more special position to negotiate pay and other things at will. Because the very existence of a business depends on them.

[tptacek]

This is like saying "the era of computer science is gone, we all just wire form fields to database columns now".

The fact that a team building the canonical wire- form- fields- to- database- columns application (if there weren't such a thing as "blogs", bug trackers would be the "hello world" of database-backed web apps) found a reason to deploy computer science is, to me, a beacon of hope that we're not all doing something that's going to be automated away in 10 years.

[firemanx]

> Building the era of custom tools just for your company, only to do specific jobs is long gone.

Isn't that the biography of 99% of the open source projects in the big data and distributed processing world? I understand they are open now, but didn't they start as custom tools just for a single company?

It seems like the "error" that Fog Creek made was to not open source Wasabi, though even that seems more like a hindsight has 20/20 vision kind of thing, as open sourcing a project is no small feat, especially to a small software company.

[applecore]

Sorry, but nine-hundred and ninety-nine times out of a thousand, it's a complete and total waste of resources to write a proprietary language or runtime enviroment to solve a non-core business problem.

[tptacek]

First: they didn't write a runtime. They used .NET/Mono as their runtime.

Second, you wrote your comment in a text box rendered by an application notorious for being written in a custom language (in fact, it seems like arc:lisp::wasabi:.net).

Third, do you have evidence to support "nine hundred ninety-nine times out of a thousand", or is that hyperbole? Have you worked on projects where people used custom languages? Your argument would be much more interesting if we could read more about your experience with it.

[to3m]

You could view their core business problem as easily creating products that as many people as possible can pay for (since that's how they make their money).

[kyllo]

And if Fog Creek were an insurance company, it would have been a correct observation.