[nmcfarl]

They lost $440 million (and amount greater than their market cap), and possibly the company, on what the world knows to be incompetence.

At some point if I couldn’t stop it - I’d be tempted to just kill the power to the server rooms, all of them. There just has to be a way to cut your losses.

[forgotusername]

I'd love to know what qualifies you to throw a word like incompetence around here. My best guess is the reason it took 45 minutes to shut it off was due to a judgement call: burn through free cash, or take out all their customers too. Bear in mind some of the largest retail brokerages in the world hang off Knight.

Their primary functions are acting as an order destination and a market-maker, for efficiency's sake an obvious conclusion would be that both functions are combined in the same software (in a market where microseconds matter). So given the choice of taking a cash hit (a potentially short term affair), or a reputation hit (a much longer term and most likely fatal affair), it's entirely possible Knight knowingly made the right decision.

It's worth note that the eventual deficit amounts to somewhere in the region of one year's net income, hardly insurmountable (and how many investment opportunities promise close to 100% return in a single year?).

Listening to the CEO on Bloomberg, it was clear that minimizing damage to customers was their primary goal (he made this point several times in the 5 minute interview), and that he appeared comfortable with the outcome.

[fauigerzigerk]

$440 million is four times their 2011 net income. I doubt the CEO is comfortable with the outcome of being on the brink of bankruptcy. He is trying to arrange a fire sale of the company as we speak.

But I think you are right that they tried to avoid an outage. The incompetence, if any, is that they apparently did not know how much money they were losing and still kept the system going. It wasn't a caclulated risk but rather an incaculable one.

I imagine it's not easy to know how much you're losing at any moment in time. They certainly knew they were building huge positions, but knowing how much they were going to lose on those positions requires an estimate of the price at which the positions can be closed (or a hedge).

What I cannot imagine is that it is common practice to leave this kind of decision to an individual's judgement call. There have to be rules for a situation like this. And there's only one sensible rule for a rogue algo racking up unknowable losses. Kill it and deal with the consequences later. Anything else is negligent.

[forgotusername]

They had >= $300m free cash as of June, that's how I arrived at the $100mish deficit.

[wpietri]

I could buy that they competently made the best of a bad situation. But I have a hard time believing that getting into the situation was the result of perfect competence.

My time writing trading software was never on the automated end of things, so I'm only modestly qualified to comment. But if I were doing the post-mortem on this one, the first thing I'd look for is middle management time pressure forcing a large release without adequate testing. And my standard for "adequate testing" would be pretty high.

If you're going to release something that can take down the company, it's worth making sure it works. In this case, they lost circa 400x the lifetime median income of a US worker. It's hard to imagine the upside that would have justified that kind of risk.

[anamax]

> In this case, they lost circa 400x the lifetime median income of a US worker.

Why is that relevant?

> It's hard to imagine the upside that would have justified that kind of risk.

Actually, it's easy to imagine such an upside. Consider 800x the lifetime median income of a US worker.

Solyndra lost far more of the US taxpayer's money. Are you really suggesting that Solyndra shouldn't have been considered because the amount of money was too large?

How about CA's high speed rail project? Are you really saying that it's a bad idea just because of the amount of money involved?

I'm not claiming that Solyndra or high-speed rail are good investments, I'm just them to demonstrate that the $500M at risk isn't a show stopper. You must consider the return.

There are lots of bets that are that large or larger. For example, every time a company sells for >$400M ....

[wpietri]

> Why is that relevant?

Because it means that there's no "we couldn't afford to do it right" excuse.

> Actually, it's easy to imagine such an upside. Consider 800x the lifetime median income of a US worker.

Double or nothing on a company that size is a stupid bet.

[anamax]

> Because it means that there's no "we couldn't afford to do it right" excuse.

No, 400x the average lifetime salary of a worker doesn't mean that.

> Double or nothing on a company that size is a stupid bet.

Wrong again. Double-or-nothing is often an extremely good bet. You're ignoring odds of each outcome.

Then again, you're just spewing soundbites and getting details right doesn't help with that.

[jussij]

> I'd love to know what qualifies you to throw a word like incompetence around here.

I'm not sure what other word you could use. Total stuff maybe?

> So given the choice of taking a cash hit (a potentially short term affair), or a reputation hit (a much longer term and most likely fatal affair), it's entirely possible Knight knowingly made the right decision.

Except thanks to their competence they have taken a massive cash hit and their reputation lies in tatters.

I suspect the damage to their reputation is so bad they will be lucky to survive.

[forgotusername]

> their reputation lies in tatters

Most of the brokerages who routed elsewhere last week were back using them as of Friday.

[Zenst]

Your right, there are so many details we dont see beyond the attention grabbing headline were everybody will naturaly go and instantly think `but all systems have a off switch, you pull the ruddy plug out` and with such headlines I'll bet most people were thinking that before even reading the article. This and how loved people in any form of finance, with HFT being one of the most loved area's. But with the blame culture in many area's of life, there will be investigations and somebody will either chuck themselfs onto a sword or somebody will be blamed and made a scapegoat. It is all down to confidence at the company from now and nomatter if they did the right things or the wrong things it is that alone that will dicate the fallout. Sad in many ways. RIM being a classic case were the media have controled the stock price, which has controled the media that control the consumers that just ends up in a deadly spiral of self forfilling doom and gloom, nomatter how well or badly they actualy do. I'm sure there are better examples but as you said, its down to company image. Nothing to do with anything else and that alone is the most important thing as a years income is not a problem if your around for more years, if it looks like you wont then it starts to spiral badly.

[rhizome]

I'm sure they would have tried pushing code in that time at least a few times.

[slurgfest]

Agreed, the incompetence of pushing changes like that to production (with such high stakes) is pretty bad... but not even having a switch to flip?

[jonhendry]

The server rooms might not be nearby. They might be located closer to the big exchanges' servers.

[jonah]

Very likely. Speed of light matters in these situations.

[iskander]

One nitpick: You can't "kill the power" on a market-making algorithm co-located with the exchange somewhere in New Jersey.

[rorrr]

It happened very quickly, after years of operations.

Technically, it can take seconds to lose that much.

What they did wrong is do all their trading with the same algorithm. Way to put all eggs in one basket.

[patdennis]

According to the article, it took them a half hour to shut it down. That's more than a few seconds.

But as a torrent of faulty trades spewed Wednesday morning from a Knight Capital Group trading program, no one at the firm managed to stop it for more than a half-hour.

[lostlogin]

It wasn't after years of operation. The article states that the system was brand new.

[sp332]

I'm pretty sure they don't do all their trading with one algorithm. What makes you think they do?