[dogma1138]

I work for the a huge financial institution, its the largest exchange by volume, we operate in multiple markets all across the globe and have our trading platforms distributed across many locations such as NYC, London and Tokyo in datacenters we own, manage and operate, our matching engine matches more daily trades than there are daily Tweets all with a guaranteed sub 50 microseconds latency with 99th percentile sitting around 20 microseconds.

We have about 4500 (up to 2 years ago it was around 2500 a bunch came in when we acquired a company nearly our size, it will probably shrink to about 3500-3000 over the next few years) employees, and about 1000 of them are engineers who develop and support a few 100’s of applications.

We have to comply to dozens of different regulations in dozens of regulatory regimes, so it’s unlikely that Twitter needs more compliance and lawyer than we do.

We are considered critical infrastructure, if we have a hiccup the world’s financial markets will be having a stroke. If we go down or worse that’s not some lost ad revenue but bilious to trillions on the line.

21K for Twitter... Lol...

[jabberwcky]

It never ceases to amaze me how easily the attitudes of finance folk so easily leak out. It seems performing a worst-case O(log N) lookup and spitting out the resulting 100 bytes over UDP counts as harder than Twitter, parent comment clearly has no idea how much work Twitter needs to perform for every tweet (URL previews much?)

Meanwhile, could have simply said you work for NYSE.

[dogma1138]

I don’t work for a stock exchange, stocks are not particularly high volume nor particularly that important for the market as many other instruments and asset classes.

Your comment shows you clearly have no idea how modern QDM and ODM markets work if you think that sending 100 bytes over UDP is all there is or even what we do, we don’t even use UDP...

[MR4D]

> could have simply said you work for NYSE

PP’s NDA probably doesn’t allow that.

[dogma1138]

No NDA just don’t see a particular reason to advertise it, it adds nothing to the argument.

[nhoughto]

An exchange like that by definition is quite focused on tight non functionals with slim functionals, aka it does a small number of things very well. Vs Uber which does a very wide set of things at a high level of consumer polish.

In my mental model of the two problems to solve, Uber is directly consumer facing to Millions of humans and that alone creates so much work in engineering that doesn’t exist when you can craft an api with sub 50 ms latency consumed by thousands of other machines.. the humans create the complexity. Solving for their problems is why you need way more people.

[dogma1138]

Order matching, order execution, credit and limit controls, reporting, multiple APIs and interfaces for order entry, market data and many other things supporting everything from legacy FIX clients coming over leased lines to modern restful web APIs and SBE over websockets, identity management with highly complicated RBAC model for traders and firms (who can trade what on behalf of whom under which conditions), highly complex and robust logging especially for regulatory requirements, maintaining alignment with regulations across the globe...

And if you think normal end users are picky you haven’t met traders that still want to use a 25 year old keyboard with non-standard ASCII codes that’s barely recognized as an HID device on your new electron based trading frontend and if you can’t accommodate for that they will leave.

So you have millions of clients all of which are adversarial and you have to maintain market fairness and integrity to ensure no manipulation is possible with trillions of dollars on the line that can be wiped off in a heartbeat and that is before you start thinking about nation state actors who would love to be able to cause major economic damage to the US, Europe and other major western economies.

[nhoughto]

Yeah the complexity is insane, millions of users is surprising tho I thought it would more be an aggregation of larger agents. I guess unicorns optimize for delivery capacity not cost

[kortilla]

> If we go down or worse that’s not some lost ad revenue but bilious to trillions on the line.

What products are exclusively traded on your exchange? If NASDAQ or NYSE goes down for half a day it’s just an article on WSJ at best. They shut down every night and for entire days on holidays. What financial exchange truly has a meaningful impact when it goes down?

[JumpCrisscross]

Equities aren’t a real economy choke point. They are also, as you observe, federated.

A more critical and delicate financial artery is the tri-party repo marker. If BoNY Mellon or JPMorgan’s repo desks went offline, it would almost certainly shut down large swathes of the American economy, prompt a credit crisis and merit urgent Federal Reserve and possibly Congressional intervention.

[kortilla]

But what is the timeline here? If JPM goes down for 2 hours it will cause a congressional intervention? That’s honestly really hard to believe and it borders in delusions of grandeur. Congress intervenes after days at a minimum of an ongoing crisis.

My point is that your strange Internet dick measuring contest of micros to match up an order book is only one tiny aspect of engineering. Your uptime numbers have lax requirements, your geographic distribution is severely limited, and your client counts (as in clients directly accessing your software) are orders of magnitude smaller than the Twitter/Uber you compare yourself to.

Operating a financial exchange it not trivial by any means, but to compare your challenges to a massive public-facing service is apples to oranges. There are likely more adversarial clients connected to twitter’s services at any given time than the peak concurrent connections to any given exchange.

[JumpCrisscross]

> * If JPM goes down for 2 hours it will cause a congressional intervention?*

It will create a crisis meriting Congressional intervention. A hiccup in the tri-party repo market automatically sends, within twenty minutes, a large fraction of the American financial system into default within twelve hours from almost anywhere on the clock.

> That’s honestly really hard to believe and it borders in delusions of grandeur

Lovely.

[kortilla]

Are you saying JPM going down means there is no alternate provider and everyone defaults in 12 hours or if the entire tri-party repo market fails? Those are not even remotely the same thing.

I would be shocked to learn that Exxon mobile will go bankrupt in half a day because one bank has an outage. It would speak volumes to how piss-poor financial networks are architected.

More realistically, JPM goes down and within an hour all of its large repo clients have already called BNY or whatever and are ready to go. Billion dollar businesses have contingency plans for headquarters being taken out by a bomb, you better believe they have a plan for their repo provider taking a shit.

> Lovely

Do you not see how ridiculous it is to suggest that the entire functioning US economy hinges on a single service run by a single company being available 24/7? If it were true, it would be a bigger terrorist target than the Whitehouse.

Apparently the entire US economy is dependent on the uptime of the work of a few engineers at one company. Better hope that information doesn’t become public or else they will be prime targets for coercion, bribes, nation state level manipulation, etc. These engineers are literally more important than Congress to keep alive and working.

[JumpCrisscross]

> Are you saying JPM going down means there is no alternate provider and everyone defaults in 12 hours

Yes. Look up how tri-party repos work [1] and then look up the Fed’s commentary on their fragility.

One of the three legs is either JPMorgan or BoNY. These are short-term but critical loans. If one of those two fails or goes offline, which practically means failure since others can’t verify the other two legs’ liquidity, large swathes of the rates, credit, futures, regular repo and stock loan markets go offline which quite literally will lead to, maybe not Exxon, but the likes GE having payments failures. You can’t “hot swap” JPMorgan to BoNY after the contract has been initiated. It would be like saying it doesn’t matter if your brokerage fails, just call someone else. Sure, for your next trade that’s fine, but in the meantime the failed broker has your assets.

> Better hope that information doesn’t become public or else they will be prime targets for coercion, bribes, nation state level manipulation, etc.

Friend is a senior IT guy at BoNY. They are regularly in touch with the Fed and FBI. It’s a known vulnerability, and there is constant scholarship and policy work on nationalising or reforming the tri-party repo market. But it’s never failed, and it’s profitable work for the two champions, so for now there are higher priorities for legislators.

Also, there are loads of these centralised pressure points in our system. In every modern financial system. Cede & Co., the Fedwire system, ADP, et cetera.

[1] https://info.bnymellon.com/rs/651-GHF-471/images/BNY_Tripart...

[dogma1138]

Yep, repo and swaps are huge, our euro repo market alone is in around quarter of trillion a day..

People don’t understand what happens when some things go down, and how it would affect them.

Futures are critical to you being able to buy milk eggs and bread at the store day after day and not having the price fluctuating on you by 100’s of % or worse those items simply being unavailable because the price either spiked too high or completely crashed so the supply is nonexistent.

Same goes for many other commodities. A cock up in the futures market can and will send ripples through the entire commodities market which will have a direct impact on the world’s supply chain.

[dogma1138]

Here is a public incident that the exchange I work for was involved in https://en.m.wikipedia.org/wiki/2010_flash_crash

It lasted half an hour, caused a trillion dollars to be wiped off the US economy and was followed by investigations and a congressional hearing.

If you think Twitter has a problem with adversarial clients connected to its network I think you don’t understand the modern financial market.

[kortilla]

> It lasted half an hour, caused a trillion dollars to be wiped off the US economy and was followed by investigations and a congressional hearing.

A bunch of things:

- that wasn’t an outage, that was actual trades taking place at significantly decreases prices. Not even close to the same category. Those same exchanges involved in that crash literally go offline every evening and on holidays/weekends and nobody cares. In fact, if the exchanges had gone completely offline during that time period, it wouldn’t have even made the news because there wouldn’t have been a place for those trades to occur.

- it didn’t cause a trillion dollars to get wiped off the economy any more than my friend transferring a share of Apple to me for $0.50 causes a trillion dollars to get wiped off of the economy.

- nobody gives a shit about a “congressional hearing”. Do you realize how many congressional hearings tech companies have been pulled in front of now? A hearing != action and if you think otherwise, there has been significantly more “congressional action” over tech companies in the last couple of years than financial companies.

> If you think Twitter has a problem with adversarial clients connected to its network I think you don’t understand the modern financial market.

Show me the public IP address of NYSE or whatever major financial exchange you want that I can connect to from any IP in the world. If exchanges were open to the world, they would get DDoSed out of existence in a heartbeat.

[dogma1138]

Derivatives and futures a lot of them are exclusive contracts not available on other exchanges, if we go down during trading hours we take the market with us.