I work for the a huge financial institution, its the largest exchange by volume, we operate in multiple markets all across the globe and have our trading platforms distributed across many locations such as NYC, London and Tokyo in datacenters we own, manage and operate, our matching engine matches more daily trades than there are daily Tweets all with a guaranteed sub 50 microseconds latency with 99th percentile sitting around 20 microseconds.
We have about 4500 (up to 2 years ago it was around 2500 a bunch came in when we acquired a company nearly our size, it will probably shrink to about 3500-3000 over the next few years) employees, and about 1000 of them are engineers who develop and support a few 100’s of applications.
We have to comply to dozens of different regulations in dozens of regulatory regimes, so it’s unlikely that Twitter needs more compliance and lawyer than we do.
We are considered critical infrastructure, if we have a hiccup the world’s financial markets will be having a stroke. If we go down or worse that’s not some lost ad revenue but bilious to trillions on the line.
It never ceases to amaze me how easily the attitudes of finance folk so easily leak out. It seems performing a worst-case O(log N) lookup and spitting out the resulting 100 bytes over UDP counts as harder than Twitter, parent comment clearly has no idea how much work Twitter needs to perform for every tweet (URL previews much?)
Meanwhile, could have simply said you work for NYSE.
I don’t work for a stock exchange, stocks are not particularly high volume nor particularly that important for the market as many other instruments and asset classes.
Your comment shows you clearly have no idea how modern QDM and ODM markets work if you think that sending 100 bytes over UDP is all there is or even what we do, we don’t even use UDP...
An exchange like that by definition is quite focused on tight non functionals with slim functionals, aka it does a small number of things very well. Vs Uber which does a very wide set of things at a high level of consumer polish.
In my mental model of the two problems to solve, Uber is directly consumer facing to
Millions of humans and that alone creates so much work in engineering that doesn’t exist when you can craft an api with sub 50 ms latency consumed by thousands of other machines.. the humans create the complexity. Solving for their problems is why you need way more people.
Order matching, order execution, credit and limit controls, reporting, multiple APIs and interfaces for order entry, market data and many other things supporting everything from legacy FIX clients coming over leased lines to modern restful web APIs and SBE over websockets, identity management with highly complicated RBAC model for traders and firms (who can trade what on behalf of whom under which conditions), highly complex and robust logging especially for regulatory requirements, maintaining alignment with regulations across the globe...
And if you think normal end users are picky you haven’t met traders that still want to use a 25 year old keyboard with non-standard ASCII codes that’s barely recognized as an HID device on your new electron based trading frontend and if you can’t accommodate for that they will leave.
So you have millions of clients all of which are adversarial and you have to maintain market fairness and integrity to ensure no manipulation is possible with trillions of dollars on the line that can be wiped off in a heartbeat and that is before you start thinking about nation state actors who would love to be able to cause major economic damage to the US, Europe and other major western economies.
Yeah the complexity is insane, millions of users is surprising tho I thought it would more be an aggregation of larger agents.
I guess unicorns optimize for delivery capacity not cost
> If we go down or worse that’s not some lost ad revenue but bilious to trillions on the line.
What products are exclusively traded on your exchange? If NASDAQ or NYSE goes down for half a day it’s just an article on WSJ at best. They shut down every night and for entire days on holidays. What financial exchange truly has a meaningful impact when it goes down?
Equities aren’t a real economy choke point. They are also, as you observe, federated.
A more critical and delicate financial artery is the tri-party repo marker. If BoNY Mellon or JPMorgan’s repo desks went offline, it would almost certainly shut down large swathes of the American economy, prompt a credit crisis and merit urgent Federal Reserve and possibly Congressional intervention.
But what is the timeline here? If JPM goes down for 2 hours it will cause a congressional intervention? That’s honestly really hard to believe and it borders in delusions of grandeur. Congress intervenes after days at a minimum of an ongoing crisis.
My point is that your strange Internet dick measuring contest of micros to match up an order book is only one tiny aspect of engineering. Your uptime numbers have lax requirements, your geographic distribution is severely limited, and your client counts (as in clients directly accessing your software) are orders of magnitude smaller than the Twitter/Uber you compare yourself to.
Operating a financial exchange it not trivial by any means, but to compare your challenges to a massive public-facing service is apples to oranges. There are likely more adversarial clients connected to twitter’s services at any given time than the peak concurrent connections to any given exchange.
> * If JPM goes down for 2 hours it will cause a congressional intervention?*
It will create a crisis meriting Congressional intervention. A hiccup in the tri-party repo market automatically sends, within twenty minutes, a large fraction of the American financial system into default within twelve hours from almost anywhere on the clock.
> That’s honestly really hard to believe and it borders in delusions of grandeur
Are you saying JPM going down means there is no alternate provider and everyone defaults in 12 hours or if the entire tri-party repo market fails? Those are not even remotely the same thing.
I would be shocked to learn that Exxon mobile will go bankrupt in half a day because one bank has an outage. It would speak volumes to how piss-poor financial networks are architected.
More realistically, JPM goes down and within an hour all of its large repo clients have already called BNY or whatever and are ready to go. Billion dollar businesses have contingency plans for headquarters being taken out by a bomb, you better believe they have a plan for their repo provider taking a shit.
> Lovely
Do you not see how ridiculous it is to suggest that the entire functioning US economy hinges on a single service run by a single company being available 24/7? If it were true, it would be a bigger terrorist target than the Whitehouse.
Apparently the entire US economy is dependent on the uptime of the work of a few engineers at one company. Better hope that information doesn’t become public or else they will be prime targets for coercion, bribes, nation state level manipulation, etc. These engineers are literally more important than Congress to keep alive and working.
Yep, repo and swaps are huge, our euro repo market alone is in around quarter of trillion a day..
People don’t understand what happens when some things go down, and how it would affect them.
Futures are critical to you being able to buy milk eggs and bread at the store day after day and not having the price fluctuating on you by 100’s of % or worse those items simply being unavailable because the price either spiked too high or completely crashed so the supply is nonexistent.
Same goes for many other commodities. A cock up in the futures market can and will send ripples through the entire commodities market which will have a direct impact on the world’s supply chain.
> It lasted half an hour, caused a trillion dollars to be wiped off the US economy and was followed by investigations and a congressional hearing.
A bunch of things:
- that wasn’t an outage, that was actual trades taking place at significantly decreases prices. Not even close to the same category. Those same exchanges involved in that crash literally go offline every evening and on holidays/weekends and nobody cares. In fact, if the exchanges had gone completely offline during that time period, it wouldn’t have even made the news because there wouldn’t have been a place for those trades to occur.
- it didn’t cause a trillion dollars to get wiped off the economy any more than my friend transferring a share of Apple to me for $0.50 causes a trillion dollars to get wiped off of the economy.
- nobody gives a shit about a “congressional hearing”. Do you realize how many congressional hearings tech companies have been pulled in front of now? A hearing != action and if you think otherwise, there has been significantly more “congressional action” over tech companies in the last couple of years than financial companies.
> If you think Twitter has a problem with adversarial clients connected to its network I think you don’t understand the modern financial market.
Show me the public IP address of NYSE or whatever major financial exchange you want that I can connect to from any IP in the world. If exchanges were open to the world, they would get DDoSed out of existence in a heartbeat.
Derivatives and futures a lot of them are exclusive contracts not available on other exchanges, if we go down during trading hours we take the market with us.
Where did you get the 100 number from? I think Nintendo is an interesting company for comparison here but I do find it hard to believe they only had 100 employees as recently as 2011.
Lyft had minuscule global presence. Uber has enormous global presence, it is a top 2 player in India , not to mention it's presence in Europe and other parts of the world.
The sheer number of languages they have provide user support and staff for driver management in these two markets alone is enormous.
Lyft has only US and Canada operations. Vast majority of Uber employees will be in customer service and operations. It it no way comparable to GitHub and Nvidia
the two largest employers are both the number one players in their market, being the number 1 player in a market may require that you invest for "first mover advantage". This would lead to a lot of failed/re-tooled initiatives over time.
Being number 2 often just means copying the number 1 player in a more capital efficient manner, product descriptions become "X did it this way, customers want it this way". In a winner takes all market, being the first mover matters a lot.
Also, diminishing returns. In many markets you can "get to the table" with a small crew, but to actually win the customer you'll need bodies almost entirely dedicated to it. And it's that sort of margin, multiplied over and over, that makes one company the undisputed n.1.
We have about 4500 (up to 2 years ago it was around 2500 a bunch came in when we acquired a company nearly our size, it will probably shrink to about 3500-3000 over the next few years) employees, and about 1000 of them are engineers who develop and support a few 100’s of applications.
We have to comply to dozens of different regulations in dozens of regulatory regimes, so it’s unlikely that Twitter needs more compliance and lawyer than we do.
We are considered critical infrastructure, if we have a hiccup the world’s financial markets will be having a stroke. If we go down or worse that’s not some lost ad revenue but bilious to trillions on the line.
21K for Twitter... Lol...