How do you handle one way gates? Clearly at each deploy there are two different versions running concurrently, and as you make changes you do so knowing that, but as the system evolves their are points in (code) time that you can't go back to. Is this not a concern because you would never roll back that far?
Yeah, rollbacks are more of adding a revert to the top of the pile so we just make sure we roll back things that can be rolled back. This is important to think about when planning deploys.
> Yeah, rollbacks are more of adding a revert to the top of the pile
Neat. It reminds me of the method used in gaming nowadays, which is to just write a "savepoint" marker in to the message stream instead of pausing the entire game to save state.
When you deploy: Will you lose requests that are currently being processed while you restart the service? Also, will a server still receive requests while the new code is being started?
Generally: no. The worker processes will finish up their current request before shutting down and being reaped. At the whole server level, Einhorn will indeed make sure that requests are still being served as the workers get shuffled out.
Actually I came to think of something but it's not an AMA kind of question but a Reddit question though. I just opted in to the profile beta literally less than an hour ago and I like it but I have a bug report [1] and a feature request / feedback [2] that I'd like to be seen by the specific people that work on the profile portion of Reddit. Who do I contact about that? Or was submitting it to /r/beta sufficient and they'll see it because I've posted it there?
> Do you find yourself browsing Reddit in your spare time or do you avoid it?
I personally still spend plenty of time reading stuff on the site, both educational (r/askhistorians, various programming subreddits) and entertainment (r/popular). I'll definitely tend to avoid the more meta subreddits when not in a work mood though.
> Do you ever find yourself spending whole days at work without being on the site because you don't want to be distracted while working on something?
Oh yeah, it definitely happens: heads down in code, doing a complex series of deploys, or just in a bunch of meetings.
There's also the times where I go to check on the site to check on something I deployed and instead get distracted by something on the front page and forget what I was doing. Oops.
I'd like to pose a follow up question; as your role at the company has progressed, do you feel your day to day (collective you, the [a]'s) is less development oriented and more social management of the different issues that arise when a community gets that large?
Actually quite the opposite. One of the benefits of being a larger company is separation of concerns. Our community team is a bunch of fantastic people that are focused on that side of things and engineering spend most of its time writing code. Obviously there will be overlap from time to time, but it's definitely way less than when we were 10 people.
Does Reddit have democratic deploys (i.e every engineer is responsible to deploy their own changes into production). Or do you guys have a system with release managers?