[gecko]

This is honestly quite a trip to read (I'm the Benjamin Pollack in the movie). I did want to make two small corrections, though:

    In one scene, she’s being interviewed on her birthday. Nobody remembered,
    so she had to buy herself a birthday hat. While she’s explaining this to the
    camera, one of her co-workers shushes her for making too much noise. On her
    birthday!

No, this coworker right here was starting to say "shit fuck shit dammit" on camera as he discovered that a stale precompiled header was getting picked up on the build box and then realized Lerone was rolling, so you're hearing me halt myself before saying a pile of profanity on film. In retrospect, Liz talking about her birthday and me suddenly cussing like a drunk sailor would've been a much better take. I regret the error.

    [Benjamin Pollack] seems to have never caught the startup bug, mainly working
    at larger, more established companies.

I worked at Fog Creek (max ~60 employees during my tenure, usually more like 40) from 2005 to 2014, Khan Academy (~120 if you count contractors) from 2015 to 2017, Spreedly (~40) from 2017 to 2018, and Bakpax (there were a dozen of us) from 2018 to 2021 (we got acquired, I wanted to stay at startups, so I didn't stay once we were bought), and another small startup briefly after that before settling at The Knot Worldwide. Yes, I'm currently at a very large company (~6500), but I'm a bit confused at how you'd come up with that summary of my career.

I honestly really enjoyed the article, though, and neither of these are exactly big errors; just some extra color I wanted to provide.

[Edit: I'm also happy to answer any questions anyone has about the movie or about that time at Fog Creek.]

[mtlynch]

Author here. Cool to see your response here!

>I'm a bit confused at how you'd come up with that summary of my career.

I worded that poorly, you're right. I meant early-stage startups at the scale of Fog Creek when you started. I've updated the post.

>No, this coworker right here was starting to say "shit fuck shit dammit" on camera as he discovered that a stale precompiled header was getting picked up on the build box and then realized Lerone was rolling, so you're hearing me halt myself before saying a pile of profanity on camera.

Ah, thanks for the context!

I watched that scene over and over trying to figure out what happened. It sounds like a "shush" but it's also weirdly stretched out. Liz seems to have heard it as a shush, as she shushes herself after. It shows Michael look over at you, but it's not clear if he was reacting to the noise.

There's an imdb trivia item[0] about how you and Liz disagree about whether it was a shush, but it's unsourced, so it was hard to put much stock in it. This is a much more satisfying answer.

I've updated the post to link to your comment here.

>I'm also happy to answer any questions anyone has about the movie or about that time at Fog Creek.

How do you feel about the movie looking back? I notice on your website that you recommend people watch it "if [they]’re feeling masochistic." Is it just the awkwardness of being the focus of a film at that age or were there things you disliked about how it came out?

What did you like/dislike about working at Fog Creek? How'd it change during your tenure?

[0] https://www.imdb.com/title/tt0813987/trivia/?item=tr0602787&...

[anildash]

One other bit of trivia, Alexis Ohanian drew one of the versions of Fog Creek’s kiwi mascot that was featured on some t-shirts and merch, though never in any of the products.

[mtlynch]

Wow, I never expected to hear from so many Fog Creek folks!

Thanks for sharing that bit of trivia, Anil. How'd that happen?

[anildash]

At that time Alexis was drawing logos or art for products he advised or knew the founders of, including most notably Hipmunk. I’m not sure if Joel asked him to draw it, or he volunteered, but it happened and then early in my time at Fog Creek, Alexis and I were catching up about some other stuff and he told me that version of the kiwi was his.

[fsckboy]

logos aren't just trademarks, they are also themselves copyrighted, and if he never explicitly gave up the rights...

[borski]

He'd have to care, which I can pretty much guarantee he doesn't.

[mtlynch]

Great bit of trivia. Thank you for sharing!

[gecko]

    There's an imdb trivia item[0] about how you and Liz disagree about whether 
    it was a shush, but it's unsourced, so it was hard to put much stock in it. 
    This is a much more satisfying answer.

Liz thought I shushed her for a long time, but I didn't even know that until years later, at which point I apologized and explained what actually happened. So I guess there was disagreement in the sense that I didn't know she thought that, but we talked that through many years ago.

I am aware of the IMDB factoid. Since they also had my bio wrong and gave me a Bacon number of two for years, I have generally just concluded IMDB doesn't care about having accurate data, and never felt a need to correct it.

    How do you feel about the movie looking back? I notice on your website that 
    you recommend people watch it "if [they]’re feeling masochistic." Is it just 
    the awkwardness of being the focus of a film at that age or were there 
    things you disliked about how it came out?

On a personal level, there's a lot I dislike. There was a lot of pressure on us to have reality-TV-show-like conflicts, which we mostly just didn't have. As a result, I end up being the antagonist in quite a few scenes (the building jump experiment is the main one), where it looks like I'm a bit of an asshole due to how things got edited together. So, normal reality TV show stuff.

(Me being awkward and arrogant also does play into it, but, like everyone else, I've grown a lot since then. Seeing how far I've come is at worst a reminder not to let myself be like that again, but usually just ends up making me feel happy I've been able to learn from my mistakes and grow. I can't say it bothers me.)

I also just kind of feel like it's a lousy movie. The soundtrack was literally written on the way to the recording studio, and you can tell. The interviews are usually not asking great questions, as much as rehashing Joel's and Paul's blogs via interviews. And there's the fact the movie is so close to being about reddit and Y Combinator right at their inception, but somehow, just...misses it.

I should tone down the "masochistic" comment so it doesn't sound like I'm bitter or hate the film. I don't. I just don't really know it's worth a watch in 2023.

    What did you like/dislike about working at Fog Creek? How'd it change during your tenure?

I learned a lot about tech, I got incredible freedom to work how I wanted on what I wanted, I had great coworkers, and I really believed in and used all of our products. They all brought me joy. That was all good. And it wasn't a grindstone like some of my friends went through at thefacebook and Google, so I had time to genuinely enjoy my hobbies and be with friends.

The dislikes are mostly just versions of me noting that working in an anarchistic environment is great only if you shout loudest and care more than the next person, but I will add that that company was so young in so many ways. We often were figuring out how to do things from base principles instead of hiring people who knew what they were doing, because we weren't sure if we knew how to tell that someone knew what they were doing. And when we started to hire our way out, we made Some Mistakes. So, things that fell out of those bits.

[mtlynch]

Thanks!

>(Me being awkward and arrogant also does play into it, but, like everyone else, I've grown a lot since then. Seeing how far I've come is at worst a reminder not to let myself be like that again, but usually just ends up making me feel happy I've been able to learn from my mistakes and grow. I can't say it bothers me.)

Yeah, there were definitely lines where my first reaction was that the interns come across badly. But I was a year or two younger than you guys at the time, and I thought about how I'd probably sound if someone asked me to talk on camera about working on software. I'm thankful that nobody did.

>The soundtrack was literally written on the way to the recording studio, and you can tell.

Oh, yeah, that does explain the soundtrack.

While I was listening to it, I felt like it was written by someone who didn't know any programmers, but they were trying to write a song that would appeal to the geeky stereotype of a programmer.

[dylan604]

> I am aware of the IMDB factoid. Since they also had my bio wrong and gave me a Bacon number of two for years, I have generally just concluded IMDB doesn't care about having accurate data, and never felt a need to correct it.

I feel like Amazon treats IMDB user submitted data just like reviews on the retail site. They just don't care. Anything goes. I've tried to get data corrected on multiple occasions for it to fall on deaf ears. Much like wikipedia where changes are reverted to previously incorrect data because of editor fiefdom turf wars. So you can't win either way.

[alexgartrell]

I had an internship at Fog Creek and would add that it was probably the most friendly and harmonious place I worked, which made it very reality-show-incompatible (and very 21-year-old-me incompatible, I wasn't asked back lol). Certainly the representation of you as an asshole was ridiculous IMO.

(Since you're answering arbitrary Fog Creek questions) In retrospect, do you think it was a mistake to make kiln hg-centric at first?

[gecko]

    (Since you're answering arbitrary Fog Creek questions) In retrospect, do you 
    think it was a mistake to make kiln hg-centric at first?

No; I think it was a mistake to not also support Subversion out-of-the-box.

Our customers were overwhelmingly Windows shops, and Git on Windows in 2007 was just unusably bad. It really would not have been a viable option. (I did look at Bazaar and Fossil, which were good players on both Windows and Unix, but neither seemed like a good fit for other reasons.) But Kiln's core value prop at the beginning was actually code review, and I think we could've found a cool way to bring in a Phabricator-like patch workflow that would've meshed just fine with Subversion and given our customers a much easier way to get access to Kiln's goodness. In that world, Mercurial would be a kind of bonus feature you could use, not the only way into Kiln. The resulting product would've been very different, mind, but I think it would've gone way better.

The other three technical mistakes we made, since you didn't ask me, were having FogBugz target .NET instead of Java (given the immaturity of Mono at the time only; I love .NET); having Wasabi compile to C# instead of IL (especially given the previous note); and having Copilot directly modifying VNC and its protocol instead of just jacketing it with a small wrapper app. These three decisions collectively slowed the company down a ton at a time when we shouldn't have let ourselves do that.

I enjoyed working with you, Alex. Glad to see you doing well!

[alexgartrell]

It was a joy to work with you as well :)

It's a bit harsh but I always feel like Fog Creek might be the cautionary tale in "what happens if you over hire for capability vs. your requirements?" I think that a less capable team would have never landed on the "let's maintain our own programming language" approach w.r.t. Wasabi.

As an aside, I do think that targeting Mono was the right thing to do for the universe, as it butterfly-effected tedu into writing weird and wonderful technical blog posts for the next ten years :p

[gecko]

    As an aside, I do think that targeting Mono was the right thing to do for
    the universe, as it butterfly-effected tedu into writing weird and wonderful 
    technical blog posts for the next ten years :p

I've never figured out whether that work broke him or was simply his muse, but I also do confess to liking the result. So not a complete loss.

[kamens]

hi alex and ben! in before HN downvotes me

[_ea1k]

I never understood the disdain for Java that I'd hear from Spolsky. It wasn't perfect, but it was certainly more cross platform than .net at the time.

[codinghorror]

I dunno, I'm with Joel on this one. Not a Java fan.

[krallja]

I was at the strategic offsite where we decided to go with .NET. Java wasn’t installed on Windows by default.

The original version of Wasabi, known as Thistle, was written in Java, by the intern in the class before Aardvark’d. It transpiled ASP to PHP.

Every intern class was named after an animal with the next consecutive letter. I don’t remember any of them except Aardvark, and I was a “B????” intern!

[c17r]

> The soundtrack was literally written on the way to the recording studio, and you can tell

You can still listen to the soundtrack on Amazon Music :-)

I bought both the DVD and the soundtrack back in the day. While it's no Grammy winner, there are a couple of good ones in there

[atesti]

>I'm also happy to answer any questions anyone has about the movie or about that time at Fog Creek

Why was FogBugz for your server discontinued silently? I think you had build something really great at the end with the change to .net and the plugin architecture (which would have made it possible to do customizations even in the cloud). Also how kiln integrated was great.

Did key people leave so that FogBugz basically stopped to be maintained anymore? Where you still around when they removed the plugins and tried to put it back and maybe switched to elastic search etc.?

[gecko]

Everything you're asking about happened after I left, so I have no idea. I didn't even know they'd discontinued FogBugz for Your Server.

[Edit: We did discontinue Kiln for Your Server while I was there. We nuked that because the support burden was monstrous, to the point we needed three extra SDETs/sales engineers purely to handle testing and on-prem bug fixes. It threatened Kiln's ability to be profitable. The FogBugz team may've made the same calculus a few years later.]

[mwcampbell]

Do you think there's anything the development team could have done to reduce the support burden, or that a team developing what we now call an on-prem product should do to minimize the support burden today? I know we have tools now that weren't available back then, e.g. containers. And .NET is open source and runs on Linux now, so that might have also helped.

[tedunangst]

Shipping a VM would have simplified things, but nobody did that, and nobody was going to download that. For Linux, we packaged everything we could, but it was still bring your own MySQL, and people had all sorts of terrible configs.

And it was kinda pre-cloud, so usually we got provisioned on some pentium ii forgotten in a closet.

[borski]

God, every time I think it's gotten better to ship appliances, it really... hasn't.

We had an on-prem solution at Tinfoil, because some of our customers needed it (gov, finance, healthcare, random big enterprise co, etc.)

We were lucky, in that we at least had Docker; or so we thought. Right up until top 5 investment bank decides to write their own orchestrator and use an internal container repository. Ugh, fine.

Oh, and also wants to use their own MySQL db? But, uh, we use Postgres... and Mongo (tech debt)... and... no?

So what did we do? We shipped a VM. You told us your VM solution of choice, we handed you a file, you set up our .ovf or .ova or whatever, it phoned home (only while setting up), got licensed, and off you went.

Debugging was miserable. We later started adding remote debugging capabilities into our contracts because the support burden was ridiculous.

Thing is, we were at least a little smart about it; single codebase, lots of feature flags, lots of internal testing (and our saas customers bled first, before VM customers), etc. I honestly might even do it again, but now there are much better solutions.

But all of this is to say: people did install VMs, and people did download them. I don't have the numbers in front of me, but I'd be mightily surprised if less than nearly 70% of our revenue (but not our customer base!) came from on-prem appliances. We sold them at a very significant premium, for obvious reasons.

[justinjlynn]

Huge respect for making that work in the first place! Would love to hear some kind of retrospective about what wa involved in that - I'm super curious.

[krallja]

We still had about a dozen developers combined on FogBugz+Kiln at the very end.

The plug-in API was very bad for performance and security, especially in a multitenant cloud application. That led a team to experimentally reimplement the entire frontend as a single-page app (code named Project Ocelot, whose shirt I am wearing today). Webhooks and a well-designed web API would have been a much tidier solution than the plug-in API.

FogBugz For Your Server had constant support costs, like Ben said, but also the application started growing all sorts of supporting services (a QueueService, an ElasticSearch cluster, Redis…) that made the existing InstallerShieid installer a huge cost to maintain.

Later, I wrote an “autosetup” script in PowerShell to help developers and support engineers onboard faster, which later became the new installer for FogBugz On-Site / Manuscript On Premises. It was designed for a single edition of Windows (Server 2012 iirc?) and SQL Server. We sold a million bucks worth of licenses for that.

[toyg]

I just assumed they followed the usual commercial trajectory: build server software, realize it's much more profitable to sell as service, discontinue on-prem sales.

[codinghorror]

Hi Ben! We'll always have London..

[gecko]

I'm happy to always have London, but I could do with forgetting most of Cambridge.

[krallja]

What about Cockfosters?