[srazzaque]

(Disclaimer that I'm working on an open source FIX engine in Lisp.)

This is a really cool project! But alas I couldn't use it as I'm not in the US, and I don't trade with Alpaca. And I feel like this plays into a negative trend I'm seeing.

The retail brokerage market right now is treading on similar 'mistakes' in institutional markets 15ish years ago. Proprietary APIs were prolific, and therefore increased switching costs and lock-in (looking at you, Reuters and Bloomberg).

On the one hand, retail brokers need to support popular programming languages and paradigms to gain traction. So, they are releasing http-based APIs, or popular language-specific libraries/bindings for their proprietary APIs. But, they are doing this instead of pushing well-publicised battle-tested standards such as FIX[1], that would normalise the playing field across brokers. So my gut feel is that they are banking on users coding against their APIs and having some lock-in effects. Notable exception is that any broker that supports cTrader will have a FIX API, because cTrader provides that for them.

The problem is: if I write a bunch of code to work against Alpaca, TD Ameritrade, or whoever, that code will likely need to be revisited for some other broker like Robinhood. Granted, market connectivity code _should_ be well abstracted from my model if I'm following good design principles. But at least with FIX, the core concepts, fields, and messages remain the same across most FIX endpoints (NOS, ExecRpt, etc), allowing for a more uniform code holistically.

[1] https://www.fixtrading.org

[daniel-levin]

I have implemented FIX for a living for a few years. It’s fragmented beyond belief. You can have different standards for different parts of the same brokerage. You have to do separate conformance tests for different asset classes at the same (big, well known, reputable) broker. The proprietary APIs like Bloomberg’s provide additional value rather than being alternatives to FIX. Make no mistake: you cannot interchange a FIX endpoint for another without making changes. The core problem is that you can impose arbitrary semantics on top of FIX messages. For example, a broker might require you to send timestamps in UTC for equities, and local time for FX, on the same FIX connection. Another might have a limit on the number of characters you can put in a ClOrdId - so GUIDs as ClOrdIds go out the window. Another big screw up is extensive use of non-ISO currency codes and the ambiguity of major and minor denomination. Is “GBp” pence? Sometimes yes! Sometimes not. Then - the most colossal screw up of them all - the reverse proxy people. They are companies that will convince management that they have a “network” and “access to liquidity”. It’s actually just a reverse proxy server that _manipulates messages in flight_! It sounds ideal, as if they will standardize your message flow so you don’t have to worry about broker specifics. In practice, it never works, and they opaquely mess with things you don’t want them to. Imagine sending an order denominated in cents, and the reverse proxy forwards it having changed the denomination to dollars. This really happened to me.

BTW, I love working on this stuff and swapping notes with other people. Drop me a line (email in profile) if you want to talk anything related to FIX!

[misja111]

I can confirm. When I was implementing FIX to communicate with some big banks, it was common practice to put everything specific for the interface as XML in a single FIX field. FIX was only used because many banks had existing infrastructure built around it, such as reverse proxies on application level that would scan the FIX messages for some standard fields and route the incoming message to the appropriate server.

[srazzaque]

Thanks for the insights, I'll certainly hit you up!

[nurettin]

FIX is just a protocol template. Vendors will make non-standard extensions to it. For example: OCO you send to CQG won't work with StoneX.

[srazzaque]

Of course, and in practice you basically always need to tweak your FIX layer when connecting to multiple brokers.

Whilst there will be differences, you can rely on a surprising amount being uniform. The general session/authentication flow and serialisation mechanisms will be largely the same. Or the use of standard fields ClOrdID, or OrderQty. Or the "general" meaning of the messages NewOrderSingle or ExecutionReport.

It's somewhat predictable where the differences are going to be: identifying securities (different codes, etc), expressing order types and algo parameters, and subtle differences to order lifecycles.

Again this is a gut feel, but I doubt that if you take 2 proprietary APIs, you'll get that level of uniformity.

[kasey_junk]

I have direct experience building the fix layers for integrating a variety of exchanges. I think FIX is a net negative. The standardization provides so little, there are so many murky edge cases in each implementation and the protocol encoding is so bad, I’d rather just program against the proprietary API. On the exchanges at least the binary APIs make so much more sense.

[CoolGuySteve]

I wish exchanges would do like Eurex and release a C header of structs representing their binary protocol.

Unambiguous, no transcription errors from an obnoxious PDF, none of FIX’s stupid design.

[kasey_junk]

It’s been an age but when I was doing connector work Eurex was so great. Everything about it technically was high quality.

[srazzaque]

I think both you and nurettin raise a very good point, in that FIX had some great aspirations, but the wiggle-room in the standards was too wide, and has resulted in, well, not a very a meaningful standard. The existence of certain companies that do nothing but aggregate and normalise broker and LP APIs is further proof of this.

Nonetheless, it'd be great to see brokers in the retail space normalise toward a uniform API. Even if that is a standardised HTTP interface with good WS support for streaming data, or a standardised binary interface. My feel is that FIX is the closest thing we have now that would get us most of the way there, even for all its warts. But of course, I'd be equally keen on anything standardised.

Edit: typo

[nurettin]

Execution report is another thing that is different between the two. You can't even list your running orders without resorting to custom tags. You can't send multiple orders without providing their custom delimiters. You have to modify login messages and insert custom tags before a message is sent. The only uniformity I am left with is the fix template of key=value\0x01 repeating N times. Nobody conforms to fix 4.2 or 5.0xxx they say they derive from it, then you go through their conformance test of 100 messages and see for yourself what changes they made.

[srazzaque]

I hear your pain... There's definitely brokers doing some strange things out there, or being overly sensitive to the order of fields. Let's not even talk about repeating groups.

> custom delimiters

Surely you mean custom _message_ delimiters, not field delimiters?

In any case I think it's a foolish endeavour to think that if a broker says "we conform to FIX4.4" to ever take that at face value. Agree that the conformance test always reveals the true extent of conformity.

But zooming out a little, despite these challenges, I'd say you're still in a better position than having N endpoints where 1 gives you a Java library with its own ridiculous threading model, another gives you a Python library that calls a HTTP endpoint (that you need to provide an OAuth token for), and yet another that only has a hosted C# interface. That's where we're headed towards.

[nurettin]

Yes I meant repeating group delimiters. What I do like about fix clients (not fix protocol) is the extra value they provide by keeping your sessions alive, tolerating hot restarts, logging in and out on schedule and managing multiple sessions. Things you have to craft when building rest clients. Other than that, I see absolutely no difference between going through a conformance test and reading rest APIs. It is basically the broker's custom designed API under the guise of a so-called industry standard fix protocol.

[srazzaque]

> What I do like about fix clients (not fix protocol) is the extra value they provide by keeping your sessions alive, tolerating hot restarts, logging in and out on schedule and managing multiple sessions.

I agree somewhat, but I think the protocol needed to come first. Can you honestly say that in its absence, multiple clients would be able to implement these behaviours in a uniform manner?

The fact that there's a concept of a "session" demarcated with LOGON/LOGOUT, sequence numbering, resend semantics, heartbeating - all this had to come from... somewhere.

Put another way: if you get 10 brokers developing their own REST/HTTP API's, what is the chance they'll normalise on a similar set of core concepts?

[peschu]

Is there a open source framework/application in go or rust based on FIX protocol? In forex quite a lot broker offer FIX apis(because MT4 is using FIX).

The use case would be to have only one application to watch/place orders simultanously on 2-3(...multiple) broker accounts. To spread risk of losing funds in case a broker goes bankrupt (or whatever reason).

[neomantra]

QuickFIX, a long-standing C++/Java FIX engine, has a Golang implementation:

https://github.com/quickfixgo/quickfix

[srazzaque]

I did not know that MT4 uses FIX!

But I can't speak for Rust or Go. If there isn't one already, there really should be...

[peschu]

Not the MT4 client but the backend seems to use fix according to: https://blackbullmarkets.com/en/platforms/fix-api-trading/

"...MetaTrader 4 (MT4) is the FIX application that BlackBull Markets uses."

[offlineguy]

I wrote a FIX engine in Common Lisp once, working for a Norwegian retail broker. Worked like a charm for many years, connected to multiple market places and brokerages. Adapting to new counterparties was certainly a bit of work, but generally far less than implementing their binary protocols. (We tried that too from time to time, usually if FIX was not an option from the exchange. Admittedly rare i later years.)

Unfortunately I wasn’t allowed to open source it, and it’s now lost in the void after a takeover. :(

Good luck to you. It’s not an impossible project. :)

[srazzaque]

Thanks! Until your reply, I was pretty sure the intersection of lispers and markets devs was close to nil. It's been mainly Java/C++ from what I've seen.

It's a shame that you couldn't open source it. If you're open to it, I'd appreciate any feedback, thoughts or code review you could offer. You can get to it (and my email) with the second link in my profile.

Edit: typo, and point about Java/C++ dominance.

[klohto]

Since you’re not from US, what trading platforms do you use? I haven’t found any with free commissions and public APIs in the EU. Well, except for Interactive Brokers.

[markvdb]

You probably won't ever see an EU broker offer free trades. The EU market is more regulated to avoid anything that smells of front running. There was a somewhat revealing interview by the people behind Flatex, a big discount broker from Germany [0].

Dutch source, do use a translator https://www.tijd.be/markten-live/nieuws/algemeen/nieuwe-eige....

"Is it easier for a pan European company to keep big US competition like Robin Hood and Ameritrade out?

Niehage: 'The companies you name are very badly positioned for Europe. Their economic model is based on two pillars. Firstly, they want to offer everything for free. But if you offer all services for free, you have to earn money with the capital in the investors' accounts. That is possible in the US, where you can still get 2% of interest. In Europe, with zero interest rates, that's very different.'

And the second pillar of their economic model?

Niehage: 'That's the high frequency traders. As far as I know, Robin Hood sells its customer orders to this kind of parties. They are prepared to pay good money for that. According to the European MiFID II-regulation, that is illegal. Those two examples show that the US and EU markets are totally different. That's why American brokers have difficulties breaking into the EU market.' "

[Slartie]

And then fintech startups like Trade Republic or Just Trade came along and...offered basically free trades (the 1€ at TR can be considered almost non-existent when comparing to the 5-10€ even at discount brokers like Flatex).

They found a third way of financing themselves: just support one marketplace which kicks back some money to them on each trade, which they in turn earn from the spread between buy and sell prices.

I thought the US predecessors of this kind of broker (Robin Hood comes to my mind first) which is still pretty new in Europe would have basically the same business model - earning some money with each trade by scraping off a few cents (or a good bunch of cents for stocks with low liquidity) from the spread?

[zingshue]

flatex also has their partners https://www.flatex.de/en/trade/premium-trading/

[OJFord]

> You probably won't ever see an EU broker offer free trades.

Freetrade, Trading212, both registered in the UK. (And I don't think 'transition period' would make a difference, but they're both years older than that.)

[markvdb]

I had a quick look at Trading212 without properly investigating their business model. They seem to route all of their stock market orders through Interactive Brokers [0].

[0] https://s3-eu-west-1.amazonaws.com/trading212.regulation/201...

[poooogles]

>Freetrade

Freetrade are now charging £3 a month for ISAs (tax free wrapped accounts) so it might be free but it's not completely free.

[OJFord]

It also has fee-less non-ISA accounts (not tax free).

[srazzaque]

I'm in Australia, so the choices here are also a bit limited. Also, I'm less interested in equities. In the CFD space, my shortlisting technique now is basically "do you offer cTrader", because as mentioned above, this means they have a FIX endpoint.

Broker offerings here seem a few "paradigm shifts" behind the US. Zero commissions are unheard of last I checked. Also, if you approach some brokers here and ask for an API or a FIX endpoint, you're either told to go away, or you're effectively ushered into a luxurious room, poured some exquisite scotch, and asked about how many millions of AUD you're looking to trade per month. And _then_ told to go away.

Edit: disambiguate "Aus" :-)

[cturner]

    > Broker offerings here seem a few "paradigm shifts"
    > behind the US. Zero commissions are unheard of last
    > I checked.

A US-style zero-commission model is not viable in the Australian landscape. The root of this is due to regulatory differences, rather than actions by the brokers.

A key to understanding the US system is something called Regulation National Market System (RegNMS). Loosely, if you are filling an order for someone, it must be at a price that is at least as good as the best price available on any US market at that time. [Anecdotally, if the NYSE receives an order, and there is a better price available for the same stock on Nasdaq at the same time, NYSE can't fill it at their inferior price.]

This creates a hyper-competitive single market. At any one time, you can know the bid/ask spread for any stock, nationally. And, it's a tight spread.

Market makers compete here. But a market maker would prefer to interact with retail orders. Because what a typical retail participant cares about is getting an immediate execution at the best price currently in the market, in order to make a long-term investment.

So a market sprung up: market-makers pay brokers to send these benign orders to them directly, without going through the public market. The market maker fills these order at a price that satisfies RegNMS. [Some of them might even distinguish themselves by giving the consumer a better price than they would have got on the public market. Your broker might send you a report with your trade saying how much "price improvement" you got compared to what you would have got at the same time on the public market.]

The reason the trade is "zero commission" is because the market maker is paying your broker for the privilege of interacting with you directly, because you don't have a view on where the market is going to be in five seconds' time.

Australia has not developed anything equivalent to RegNMS. Without that, there is no foundation for the zero-commission trades (there is no national best price to protect consumers with). In Australia, the typical situation is retail orders being sent directly to the book on the ASX (a pseudo-monopoly exchange), where they are directly interacting with the most sophisticated houses.

If you google, you can find an ASIC paper trumpeting that Payment for Order Flow has been banned. In my view, the real story is a lack of innovation by Australia's regulators.

[pedalpete]

Is IB "Interative Brokers"? I'm trying to figure out if Aus is Australia or Austria. I'm in Australia, and have been looking at who to go with. Almost signed up with CBA last week, but definitely more expensive to what I'm used to in North America.

[srazzaque]

Yep, IB is Interactive Brokers. And I hear you about the brokerage rates here!

[ag56]

https://www.freetrade.io/

[hnarn]

Not sure about API but there’s Degiro.

[nelaboras]

IB doesn't seem to be commission free though.

[phyalow]

IBKR Lite is in the US, IBKR Pro in Aus etc is not.

[mesika]

I think IBKR Lite don’t provide api trading.

[Random_ernest]

Probably not the question you had in mind, but what Lisp and what made you choose it?

[srazzaque]

Common Lisp on SBCL.

I lived on the JVM for quite some time (and still do), and I wanted to explore other ecosystems out of sheer curiosity. Got to playing with Lisp/SBCL, some initial tests showed it was pretty quick, and was impressed that I could disassemble functions in the REPL and iterate rapidly. Then a while later... I noticed there wasn't a FIX library at around the same time I was exploring creating my own trading models.

So I can't say there was a lot of analysis across the different Lisps, etc. Had the coin flipped a little differently a while back this might have been in Racket. Also the choice of language in the book Professional Automated Trading[1] did influence me somewhat.

[1] https://www.wiley.com/en-us/Professional+Automated+Trading%3...

[thidr0]

Do you recommend that book and/or any others? Interested in automated trading + functional languages

[srazzaque]

I read it quite a while back, so memory is a little hazy.

It's one of the few books (perhaps the only one I've found) that approaches this subject matter from the perspective of an engineer wanting to architect their own system from the ground up, and potentially work for themselves or in a firm. But, going in, you should know that it focusses less on trading techniques and more on software architecture. What data structures you might use, and one potential approach that's laid out in the book.

Whilst some areas in the book are light (for example, the performance section), I'd recommend it if you're interested in ground-up building a trading system. If you work in the space already, you'll know the parts that are misaligned with majority of systems out there (choice of language being one). If you don't, and are looking to enter the space, it'll increase familiarity with core concepts (for instance, before I read this book, I had no distinction between "model" and "algo").

It's also refreshing that it stands out from the pack of books that aim to make you zillions of dollars just by using "these 10 key trading secrets!".

[thidr0]

Excellent. That sounds like something I'd really enjoy. I don't work in the space, but I'm more interested in the trading systems than the algos, primarily. After all, I'm a little skeptical about my ability to beat the PhDs with their institutional nukes while I'm over here sharpening a dagger.

[paladfn]

I would recommend "Professional Automated Trading: Theory and Practice" [0]. The code is here [1].

[0]: https://www.amazon.com/Professional-Automated-Trading-Theory...

[1]: https://github.com/wzrdsappr/trading-core

[Amorymeltzer]

For the US and Robinhood folks: there's a fairly effective project that allows you to do something similar: https://github.com/bcwik9/robinhood-on-rails

I used it a bit for fun, seemed legit and worked fine.

[mellosouls]

How is it similar; the whole point of the parent comment is independence from that walled-garden approach - it sounds like precisely the opposite to me?

[loblollyboy]

Robinhood has a website tho

[jetti]

Has the FIX spec been free and open for a long time? I was going to create my own FIX engine a couple years back but for some reason I remember having to pay for the spec and it wasn't cheap.

[zelly]

The proprietary API is guaranteed to be slower. The brokers must send order flow as FIX to the exchange. This adds latency of however long it takes the broker's software to translate to FIX. I don't think there are any successful quant strategies that don't rely on speed, so basically anything with a proprietary API is solely for amateurs.

[oarsinsync]

> I don't think there are any successful quant strategies that don't rely on speed

There are plenty of successful quantitative algorithms that are in use by non HFT firms

> so basically anything with a proprietary API

If you’re not a direct market participant, colocated in the exchange, you’re not in a position to be talking about speed at all. Also most exchanges optimising for speed are not using FIX.

> is solely for amateurs

You’re not necessarily an amateur, you’re just leaving money on the table for someone else as you’ve not optimised that area of your trading infra yet.

[smabie]

I don't think you understand what quant finance is about. Almost all quant strategies don't rely on speed. In fact, HFT is usually considered a totally different space than investing (and therefore quant investing).

[egwor]

> The proprietary API is guaranteed to be slower. Yeah, most of the time that's not the case. Binary encoding/decoding is often much faster than FIX. Look at ITCH/OUCH, SBE and FastFix.

[getlawgdon]

Actually the majority of quant trading isn't dependent on speed.