[atesti]

>I'm also happy to answer any questions anyone has about the movie or about that time at Fog Creek

Why was FogBugz for your server discontinued silently? I think you had build something really great at the end with the change to .net and the plugin architecture (which would have made it possible to do customizations even in the cloud). Also how kiln integrated was great.

Did key people leave so that FogBugz basically stopped to be maintained anymore? Where you still around when they removed the plugins and tried to put it back and maybe switched to elastic search etc.?

[gecko]

Everything you're asking about happened after I left, so I have no idea. I didn't even know they'd discontinued FogBugz for Your Server.

[Edit: We did discontinue Kiln for Your Server while I was there. We nuked that because the support burden was monstrous, to the point we needed three extra SDETs/sales engineers purely to handle testing and on-prem bug fixes. It threatened Kiln's ability to be profitable. The FogBugz team may've made the same calculus a few years later.]

[mwcampbell]

Do you think there's anything the development team could have done to reduce the support burden, or that a team developing what we now call an on-prem product should do to minimize the support burden today? I know we have tools now that weren't available back then, e.g. containers. And .NET is open source and runs on Linux now, so that might have also helped.

[tedunangst]

Shipping a VM would have simplified things, but nobody did that, and nobody was going to download that. For Linux, we packaged everything we could, but it was still bring your own MySQL, and people had all sorts of terrible configs.

And it was kinda pre-cloud, so usually we got provisioned on some pentium ii forgotten in a closet.

[borski]

God, every time I think it's gotten better to ship appliances, it really... hasn't.

We had an on-prem solution at Tinfoil, because some of our customers needed it (gov, finance, healthcare, random big enterprise co, etc.)

We were lucky, in that we at least had Docker; or so we thought. Right up until top 5 investment bank decides to write their own orchestrator and use an internal container repository. Ugh, fine.

Oh, and also wants to use their own MySQL db? But, uh, we use Postgres... and Mongo (tech debt)... and... no?

So what did we do? We shipped a VM. You told us your VM solution of choice, we handed you a file, you set up our .ovf or .ova or whatever, it phoned home (only while setting up), got licensed, and off you went.

Debugging was miserable. We later started adding remote debugging capabilities into our contracts because the support burden was ridiculous.

Thing is, we were at least a little smart about it; single codebase, lots of feature flags, lots of internal testing (and our saas customers bled first, before VM customers), etc. I honestly might even do it again, but now there are much better solutions.

But all of this is to say: people did install VMs, and people did download them. I don't have the numbers in front of me, but I'd be mightily surprised if less than nearly 70% of our revenue (but not our customer base!) came from on-prem appliances. We sold them at a very significant premium, for obvious reasons.

[justinjlynn]

Huge respect for making that work in the first place! Would love to hear some kind of retrospective about what wa involved in that - I'm super curious.

[krallja]

We still had about a dozen developers combined on FogBugz+Kiln at the very end.

The plug-in API was very bad for performance and security, especially in a multitenant cloud application. That led a team to experimentally reimplement the entire frontend as a single-page app (code named Project Ocelot, whose shirt I am wearing today). Webhooks and a well-designed web API would have been a much tidier solution than the plug-in API.

FogBugz For Your Server had constant support costs, like Ben said, but also the application started growing all sorts of supporting services (a QueueService, an ElasticSearch cluster, Redis…) that made the existing InstallerShieid installer a huge cost to maintain.

Later, I wrote an “autosetup” script in PowerShell to help developers and support engineers onboard faster, which later became the new installer for FogBugz On-Site / Manuscript On Premises. It was designed for a single edition of Windows (Server 2012 iirc?) and SQL Server. We sold a million bucks worth of licenses for that.

[toyg]

I just assumed they followed the usual commercial trajectory: build server software, realize it's much more profitable to sell as service, discontinue on-prem sales.