[skizm]

Slightly off topic, but if a bug like this is discovered does the engineer who wrote it get notified?

It would be funny to have a sort of wall of shame for that week or something else internally. Or you could even go as far as making that engineer pay for the bug bounty (that's a bit much though). Anyone have any experience as to what happens on Google's end besides the obviously patching of the bug and paying of the fine?

[Afforess]

A wall of shame sounds amusing at first blush, but it would quickly become a source of a lot of negativity and unhappiness. Yes, developers need to be aware of bugs, and learn from mistakes, but intentional harassment seems a step too far. I know I've written thousands of bugs.

[ketralnis]

A friend that used to work at Apple once told me a story of their department having a big banner over the desk of the last person to broke the build. Eventually a frustrated recipient of the banner solved the problems causing it to be brittle in the first place, and as a result had the banner himself for months. A rather unexpected disincentive to solve the problem

[skizm]

I agree it can definitely turn counter-productive quickly. On the other hand, if not a wall of shame, I would like to know what happened if it was my code.

[eieio]

It might be funny but I can't imagine that shaming engineers like that would be very productive. Anyone talented enough to be working at a place like Google is likely going to be plenty embarrassed without a "wall of shame" to make sure everyone knows who screwed up.

Not a Google employee but I'd imagine they'd have to do some kind of writeup about what happened/how future errors like this will be prevented.

[RKearney]

But whose fault is it? Is it the programmers fault for writing the bug? Or is it the QA's fault for missing the bug? Or was it the leads fault for signing off on buggy code?

You can point the blame at a lot of people, but in the end it's highly unproductive and a waste of time. You fix the bug and move on. If programming teams played the blame game every time a bug came up, it would just slow everyone down.

[skizm]

All very good questions. I was wasn't thinking about it as the blame game though, but I do see how it could quickly become that.

I was thinking, firstly, along the lines of having some ammo to make fun of people with (insult based humor is the basis for most of my relationships with people). Also, on a more practical note, as a developer myself I would like to know when my code breaks and why.

[ChuckMcM]

Actually there was (probably still is) an internal newsletter from the security team on the latest stuff that anyone who does anything customer facing should read. I really enjoyed reading it, both the writing was creative (engaging as opposed to obfuscating :-) and the topics sort of like logic puzzles.

[jrockway]

It's not particularly constructive to think of individuals as being responsible for single bugs. Bugs are random events, and their probability increases as people and processes become complacent.

A good way to react to this bug is to come up with ideas for reducing the possibility of future bugs: static analysis tools, making code reviews easier, and so on. One might also think up ways to lower the impact of future bugs; make session cookies unavailable to JavaScript, propose new standards for the web, etc.

It's important to think of it from a statistical standpoint. Given two equally skilled developers, the one that implements more features is more likely to be involved in a production incident. If we punish people for bugs, we're punishing productivity in addition to sloppiness. That sets the incentives incorrectly. It isn't even a good idea to blame one person: if you scare one person into compliance, you still have the 29,999 other SWEs that aren't scared into compliance. Much better to develop tools that make bugs easier to spot, because every hour you spend doing that helps 29,999x as many people.

[arkem]

Disclaimer: I wasn't involved in this particular bug at all so I'm speaking generally.

We definitely don't play the blame game but we do keep track of bug statistics so we know where best to spend our time and effort. It can be useful to know that projects using framework X have more issues than framework Y or that maybe we should arrange to run some security classes at office Z.

Bugs like this one are fixed with the help of the product team, they're usually the ones writing and pushing the fix (since they know the project best) and it's a good way to get some practical security experience spread around the organization (and increase awareness).

We do write post-mortems for serious issues to delve into the root cause and to help stop it from happening again. We have a lot of initiatives in place to improve the security of our products overall through both awareness raising (training, newsletters, security puzzles) and technology changes (scanners, static analysis, framework hardening).

P.S We're always looking for security people to join us at Google (send me your resume, email in profile) or by bug hunting for bugs to submit to our vulnerability reward program.

[btgeekboy]

Praise your employees in public. Correct them in private.

People like to be lauded for their accomplishments, but nobody wants to be known as "that guy who wrote a severe XSS bug."

Build babysitters are a bit of an exception, since it's not the end of the world if the current build is broken - another commit, and the problem's resolved. Public XSS though... that's more severe.

[skybrian]

That's needlessly vicious. You don't need a wall of shame. Just report the bug back to the original programmer and they'll feel bad enough as it is.

Also, generally speaking, it's not one person's fault. At the very least they also have a reviewer who should have caught it.

What actually happens in a healthy organization is that someone writes a postmortem going through all the steps leading to the bad outcome, along with a plan to make fixes at multiple levels to make sure nothing like it happens again.

[astral303]

I hope that the engineer(s) that wrote it get notified, so that they get the feedback on their decisions. If I write code with bugs that make it to production, I'd sure would like to know, even if I'm not the one fixing it.

(This is the major reason why I think having sustaining/continuing engineering departments in software companies is a bad idea).