[svdad]

What I wonder, following this story this week, is how the software quality controls at a place like Knight compare with those for life-critical systems like those in, e.g., aviation.

On one hand, you'd think the QA in finance would be pretty solid, considering that the survival of the company could be at stake (witness Knight). On the other hand, I have a feeling that even there, people just don't take it that seriously.

Would love to hear from anyone with more experience writing software for these industries.

[ewood]

I have experience in HFT, there are similarities to market making and I have plenty of colleagues who've worked in market making. Just like any company the culture is largely dependent on those in charge. Founders of these companies fall into three buckets - traders, techies, and mathematicians/physicists - and quality control will generally be a function of the founder mix. Mostly techies: strong software culture, unit testing, realtime monitoring. Mostly scientists: strong algos, software that works, monitoring varies. Mostly traders: risk v reward is the driver, software quality is unimportant unless it affects short-term profit, monitoring is unimportant - dollars and cents are sufficient.

Obviously every firm's goals are driven by the goals of those in control. In the case of Knight they are largely a trader driven firm that has arrived late to the algo party. They were looking to get ahead by being one of the first market makers on the NYSE's new retail order matching system and probably cut some corners to get there. From a risk v reward perspective it probably looked like a good bet - with no major competitors customers would flood in and any bugs could be ironed out in live. Unfortunately the 'fat tail' (http://en.wikipedia.org/wiki/Fat_tail) struck and it may have sunk their company.

For a closer look at what went wrong see http://www.zerohedge.com/news/what-happens-when-hft-algo-goe...

[jriley]

Thanks for link. Given your experience, do you have any thoughts about a small circuit-breaker on every security that trips for, say any 3-standard deviation event?

In a quick Google search, it seems circuit breakers do not kick in for first 15 minutes of trading: http://www.nytimes.com/2012/08/02/business/unusual-volume-ro...

[ewood]

In derivative markets such as futures there are predefined price and volume limits (in the jargon 'limit up' and 'limit down' - ie: max up and down movements.) These limits exist primarily to prevent market manipulation or cornering a market (buying the entire supply of a commodity) but have also been triggered by trading around events such as the Japan earthquake. Doing a quick search it seems that there are plans to trial these controls on equity markets (http://blogs.law.harvard.edu/corpgov/2012/06/13/limit-up-lim...)

The problem here though was that while some stocks had dramatic price movements that might have triggered a limit control, more heavily traded stocks were able to absorb the additional volume and the price did not move significantly. Knight were not doing anything outside of normal bands, they were buying normal volumes of stocks close to the current market price and selling close to the current price. What they were doing was illogical in a profit sense because they were buying at a high price and selling at a lower price and thus immediately losing money. I think it would be very difficult for an exchange to trap this kind of problem.

In all the issue of determining 'normal' trading is very difficult as markets tend to be much noisier than you might expect. The majority of trading occurs near the open and closes of major markets (Hong Kong, London, New York) or data releases (eg US Unemployment) so large spikes in volume and price are a regular occurrence. In equities this is even more difficult as smaller stocks will tend to be more volatile and profit reporting season increases this volatility even further. Markets are ruled by fear and greed, and falsely triggering a limit may cause larger issues than it solves.

[teyc]

Perhaps one way to mitigate this is by wrapping the calls to another system such that all trading is supervised.

Another technique is to provide API keys in the wrapper so that test programs will not have the keys to a live system.

The real problem is that the risk of these test systems have not been sufficiently identified or recognized. We are all too busy creating mock systems instead of devoting sufficient oversight to the development of test software.

[jriley]

Good insight.

Made me wonder what would happen to volume if markets were open 24/7...but I'll leave that thought for another day.

[izaidi]

Unlike high-frequency trading, aviation is highly regulated. In the United States the FAA specifies pretty detailed development standards for avionics software (e.g., DO-178B: http://en.wikipedia.org/wiki/DO-178B). We're unlikely to see similarly strict requirements for financial software anytime soon.

[wpietri]

Having talked with people who write life-critical code, the regulation isn't really what makes it safe. Safety comes from good engineering.

The regulation just makes it much harder to bring an unsafe product to market, and makes it clearer who to blame when people die.

[wglb]

But don't you think the existence of regulation influences the culture?

[toomuchtodo]

Penalties influence the culture. As we all know, the first lesson in economics is that incentives matter.

Sometimes, the right people aren't being incentivized to do the right thing.

[Retric]

much harder to bring an unsafe product to market I would assume that's the point of the regulation in the first place? Nothing guarantees great software, but say requiring companies to pay for independent 3rd party testing adds significant barriers.

[jimmytucson]

I don't blame anybody who thinks it's not as regulated as it should be but you might be surprised at how extensive the current requirements are. Storing ~200 million daily transactions in order to fulfill arbitrarily complex reporting specifications from SEC, FINRA, internal audit (rogue trading detection, etc.), not to mention analytics for other trading systems... Oracle/IBM/whatever enterprise big data provider gets the contract must be sending one gigantic fruit basket around Christmas time.

EDIT: Here's a little light reading for you http://finra.complinet.com/en/display/display_main.html?rbid... (or you could just take sandpaper and rub it against your brain)

[AncientPC]

I can't find the links at the moment, but some HFT guys have responded to relevant Slashdot articles over the years.

From what I remember from those accounts, traders often like to tweak algorithms at the last second and there is little to no QA before changes gets pushed live.

It's sort of a byproduct of the necessity for speed. Algorithms are quite complex, and even running a basic test suite that takes a few minutes may be deemed unnecessary.

[njharman]

> considering that the survival of the company could be at stake

It's at stake either way.

QA is not risk free. Time spent in QA is opportunity cost lost. Many things are very time to market sensitive. One must balance "perfect" against "shipped".

This is even more true in (electronic) financial industry.

[thibaut_barrere]

I can tell first-hand that it depends on the places: some have fairly clean tests including fuzzy testing on trading automation, while other have, well, "fix it when it fails in production".