[mrfox321]

At a big co I worked at, the lack of consistency between trading systems caused money to (dis)appear (into)out of thin air.

Prior to one of these hiccups, I hypothesized, given how shitty the codebase was, that they must be tracking this stuff poorly.

This led to an argument with my boss, who assumed things magically worked.

Days later, we received an email announcing an audit one one of these accounting discrepancies.

JPMC proposed using crypto, internally, to consistently manage cash flow.

Not sure if it went anywhere.

[HolyLampshade]

At all of the exchanges and trading firms I’ve worked with (granted none in crypto) one of the “must haves” has been a reconciliation system out of band of the trading platforms. In practice one of these almost always belongs to the risk group (this is usually dependent on drop copy), but the other is entirely based on pcaps at the point of contact with every counterparty and positions/trades reconstructed from there.

If any discrepancies are found that persist over some time horizon it can be cause to stop all activity.

[ajb]

Wait, pcap as in wireshark packet capture?

[tnlnbn]

I'm not the commenter, but yes, often trading firms record all order gateway traffic to from brokers or exchanges at the TCP/IP packet level, in what are referred to as "pcap files". Awkwardly low-level to work with, but it means you know for sure what you sent, not what your software thought it was sending!

[pclmulqdq]

The ultimate source of truth about what orders you sent to the exchange is the exact set of bits sent to the exchange. This is very important because your software can have bugs (and so can theirs), so using the packet captures from that wire directly is the only real way to know what really happened.

[generic92034]

But then the software capturing, storing and displaying the packets can also have bugs.

[bostik]

Among all the software installed in a reputable Linux system, tcpdump and libpcap are some of the most battle tested pieces one can find.

Wireshark has bugs, yes. Mostly in the dissectors and in the UI. But the packet capture itself is through libpcap. Also, to point out the obvious: pcap viewers in turn are auditable if and when necessary.

[kortilla]

They can, but it’s far less likely to be incorrect on the capture side. They are just moving bytes, not really doing anything with structured data.

Parsing the pcaps is much more prone to bugs than capturing and storing, but that’s easier to verify with deserialize/serialize equality checks.

[chairmansteve]

The result of bitter lessons learnt I'm sure. Lessons the fintechs have not learned.

[baq]

That makes sense - but it's still somewhat surprising that there's nothing better. I guess that's the equivalent of the modern paper trail.

[HolyLampshade]

It’s the closest to truth you can find (the network capture, not the drop copy). If it wasn’t on the network outbound, you didn’t send it, and it’s pretty damn close to an immutable record.

[ajb]

It makes sense. I'm a little surprised that they'd do the day to day reconciliation from it but I suppose if you had to write the code to decode them anyway for some exceptional purpose, you might as well use it day to day as well.

[thomasjudge]

The storage requirements of this must be impressive

[bostik]

Storage is cheap, and the overall figures are not that outlandish. If we look at a suitable first page search result[0], and round figures up we get to about 700 GB per day.

And how did I get that figure?

I'm going to fold pcap overhead into the per-message size estimate. Let's assume a trading day at an exchange, including after hours activity, is 14 hours. (~50k seconds) If we estimate that during the highest peaks of trading activity the exchange receives about 2M messages per second, then during more serene hours the average could be about 500k messages per second. Let's guess that the average rate applies 95% of the time and the peak rate the remaining 5% of the time. That gives us an average rate of about 575k messages per second. Round that up to 600k.

If we assume that an average FIX message is about 200 bytes of data, and add 50 bytes of IP + pcap framing overhead, we get to ~250 bytes of transmitted data per message. At 600k messages per second, 14 hours a day, the total amount of trading data received by an exchange would then be slightly less than 710GB per day.

Before compression for longer-term storage. Whether you consider the aggregate storage requirements impressive or merely slightly inconvenient is a more personal matter.

0: https://robertwray.co.uk/blog/the-anatomy-of-a-fix-message

[tetha]

And compression and deduplication should be very happy with this. A lot of the message contents and the IP/pcap framing overheads should be pretty low-entropy and have enough patterns to deduplicate.

It could be funny though because you could be able to bump up your archive storage requirements by changing an IP address, or have someone else do that. But that's life.

[oblio]

Why? They're not streaming 4k video, it's either text protocol or efficient binary protocols.

[w23j]

I would also really like to know that!

It generally seems to be a thing in trading: https://databento.com/pcaps

There is also this (though this page does not specify what pcap means): https://www.lseg.com/en/data-analytics/market-data/data-feed...

[hnbear]

Look up Corvil devices by Pico.

Commonly used in finance.

https://www.pico.net/corvil-analytics/

[hnbear]

Typically not a literal pcap. Not just wireshsrk running persistently everywhere.

There are systems you can buy (eg by Pico) that you mirror all traffic to and they store it, index it, and have pre-configured parsers for a lot of protocols to make querying easier.

Think Splunk/ELK for network traffic by packet.

[cjalmeida]

Except it is literal “pcap” as they capture all packets at layer 3. I don’t know the exact specifications of Pico appliances, but it would not surprise me they’re running Linux + libpcap + some sort of timeseries DB

[hnbear]

Well, probably, but I meant more like it's not typically someone running tcpdump everywhere and someone analyzing with Wireshark, rather than a systems configured to do this at scale across the desktop.

[kelnos]

I don't think that's what anyone was assuming. A "pcap" is a file format for serialized network packets, not a particular application that generates them.

[hn_version_0023]

The Corvil devices used by Pico have IME largely been replaced by Arista 7130 Metamux platforms at the capture “edge”

[HolyLampshade]

Which is great for the companies that have made the switch because those corvils were truly terrible.

[HolyLampshade]

Looks like tnlnbn already answered, but the other benefit to having a raw network capture is often this is performed on devices (pico and exablaze just to name two) that provide very precise timestamping on a packet by packet basis, typically as some additional bytes prepended to the header.

Most modern trading systems performing competitive high frequency or event trades have performance thesholds in the tens of nanos, and the only place to land at that sort of precision is running analysis on a stable hardware clock.

[Loic]

I suppose Pre-Calculated Aggregated Positions, but I am not an expert in the field.

[SnorkelTan]

Looking at the order messages sent to and received from another trading system was not uncommon when I worked in that neck of the woods

[chairmansteve]

The crypto firms are moving fast and breaking things. No need for that kind of safety shit, right? Would slow things down. Reminds me of Boeing.

[nobodyandproud]

So is this capture used to reconstruct FIX messages?

[HolyLampshade]

Yeah, FIX or whatever proprietary binary fixed-length protocols (OUCH or BOE for example) the venue uses for order instructions.

Some firms will also capture market data (ITCH, PITCH, Pillar Integrated) at the edge of the network at a few different cross connects to help evaluate performance of the exchange’s edge switches or core network.

[phire]

Fun fact, centralized crypto exchanges don't use crypto internally, it's simply too slow.

As a contractor, I helped do some auditing on one crypto exchange. At least they used a proper double-entry ledger for tracking internal transactions (built on top of an SQL database), so it stayed consistent with itself (though accounts would sometimes go negative, which was a problem).

The main problem is that the internal ledger simply wasn't reconciled with with the dozens of external blockchains, and problems crept in all the time.

[oblio]

> Fun fact, centralized crypto exchanges don't use crypto internally, it's simply too slow.

I know you're not arguing in their favor, just describing a reality, but the irony of that phrase is through the roof :-)))

Especially the "centralized crypto".

[phire]

Yeah, that fact alone goes a long way to proving there is no technical merit to cryptocurrencies.

The reason they are now called "centralised crypto exchanges" is that "decentralised crypto exchanges" now exist, where trades do actually happen on a public blockchain. Though, a large chunk of those are "fake", where they look like a decentralised exchange, but there is a central entity holding all the coins in central wallets and can misplace them, or even reverse trades.

You kind of get the worst of both worlds, as you are now venerable to front-running, they are slow, and the exchange can still rug pull you.

The legit decentralised exchanges are limited to only trading tokens on a given blockchain (usually ethereum), are even slower, are still vulnerable to front-running. Plus, they spam those blockchains with loads of transactions, driving up transaction fees.

[naasking]

> JPMC proposed using crypto, internally, to consistently manage cash flow.

Yikes, how hard is it to just capture an immutable event log. Way cheaper than running crypto, even if only internally.

[imglorp]

Harder than you'd think, given a couple of requirements, but there are off the shelf products like AWS's QLDB (and self hosted alternatives). They: Merkle hash every entry with its predecessors; normalize entries so they can be consistently hashed and searched; store everything in an append-only log; then keep a searchable index on the log. So you can do bit-accurate audits going back to the first ledger entry if you want. No crypto, just common sense.

Oddly enough, I worked at a well known fintech where I advocated for this product. We were already all-in on AWS so another service was no biggie. The entrenched opinion was "just keep using Postgres" and that audits and immutability were not requirements. In fact, editing ledger entries (!?!?!?) to fix mistakes was desirable.

[to11mtm]

> The entrenched opinion was "just keep using Postgres" and that audits and immutability were not requirements.

If you're just using PG as a convenient abstraction for a write-only event log, I'm not completely opposed; you'd want some strong controls in place around ensuring the tables involved are indeed 'insert only' and have strong auditing around both any changes to that state as well as any attempts to change other state.

> In fact, editing ledger entries (!?!?!?) to fix mistakes was desirable.

But it -must- be write-only. If you really did have a bug fuck-up somewhere, you need a compensating event in the log to handle the fuck-up, and it better have some sort of explanation to go with it.

If it's a serialization issue, team better be figuring out how they failed to follow whatever schema evolution pattern you've done and have full coverage on. But if that got to PROD without being caught on something like a write-only ledger, you probably have bigger issues with your testing process.

[rdpintqogeogsaa]

Footnote to QLDB: AWS has deprecated QLDB[1]. They actually recommend using Postgres with pgAudit and a bunch of complexity around it[2]. I'm not sure how I feel about such a misunderstanding of one's own offerings of this level.

[1] https://docs.aws.amazon.com/qldb/latest/developerguide/what-...

[2] https://aws.amazon.com/blogs/database/replace-amazon-qldb-wi...

[imglorp]

Yeah. I'm surprised it didn't get enough uptake to succeed, especially among the regulated/auditable crowds, considering all the purpose built tech put into it.

[naasking]

I think you're forgetting how many businesses are powered by Excel spreadsheets. This solution seems too advanced and too auditable.

[baq]

I'll just leave that here for no particular reason at all:

https://www.sec.gov/enforcement-litigation/whistleblower-pro...

[voidfunc]

Better hurry, Elon is gonna dismantle the SEC in about 45 days

[fragmede]

Importantly, the SEC is empowered to give 10-30% of the money siezed via whistleblowing too the whistle blower.

[chipsa]

> Merkle hash every entry with its predecessors; normalize entries so they can be consistently hashed and searched; store everything in an append-only log;

Isn’t this how crypto coins work under the hood? There’s no actual encryption in crypto, just secure hashing.

[limit499karma]

Theoretically they even have a better security environment (since it is internal and they control users, code base and network) so the consensus mechanism may not even require BFT.

[hooverd]

It's all merkle trees under the hood. I feel like the crypto coin stuff has overshadowed the useful bits.

[trog]

Is a Merkle tree needed or is good old basic double ledger accounting in a central database sufficient? If a key requirement is not a distributed ledger then it seems like a waste of time.

[Onavo]

Merkle tree is to prevent tampering, not bad accounting practices

[nly]

It only prevents tampering if the cost of generating hashes is extremely high.

Internally in your company you're not going to spend millions of $'s a year in GPU compute just to replace a database.

[xorcist]

"Prevents tampering" lacks specificity. git is a blockchain that prevents tampering in some aspects, but you can still force push if you have that privilege. What is important is understand what the guarantees are.

[limit499karma]

? If I use something like Blake3 (which is super fast and emits gobs of good bits) and encode a node with say 512 bits of the hash, you are claiming that somehow I am vulnerable to tampering because the hash function is fast? What is the probable number of attempts to forge a document D' that hashes to the very same hash? And if the document in structured per a standard format, you have even less degrees of freedom in forging a fake. So yes, a Merkel tree definitely can provide very strong guarantees against tampering.

[oconnor663]

Fwiw, increasing the BLAKE3 output size beyond 256 bits doesn't add security, because the internal "chaining values" are still 256 bits regardless of the final output length. But 256 bits of security should be enough for any practical purpose.

[benlivengood]

Certificate transparency logs achieve tamper-resistance without expensive hashes.

[agentultra]

Write-Once, Read Many drives also prevent tampering. Not everything needs crypto.

[chaboud]

In a distributed setting where a me may wish to join the party late and receive a non-forged copy, it’s important. The crypto is there to stand in for an authority.

[trog]

> In a distributed setting where a me may wish to join the party late and receive a non-forged copy, it’s important. The crypto is there to stand in for an authority.

Yeh, but that's kinda my point: if your primary use case is not "needs to be distributed" then there's almost never a benefit, because there is always a trusted authority and the benefits of centralisation outweigh (massively, IMO) any benefit you get from a blockchain approach.

[jchanimal]

We launched Fireproof earlier this month on HN. It’s a tamperproof Merkle CRDT in TypeScript, with an object storage backend for portability.

See our Show HN: https://news.ycombinator.com/item?id=42184362

We’ve seen interest from trading groups for edge collaboration, so multi-user apps can run on-site without cloud latency.

[hluska]

What disrespectful marketing. We don’t care that you use Merkle trees because that’s irrelevant. I guess I can add Fireproof to my big list of sketchy products to avoid. It’s embarrassing.

[jchanimal]

I figured the responses would be more interesting. Questions about CRDT guarantees etc.

Perhaps worth seeding the convo with a remark about finality.

[hluska]

While your intentions may have been around discussion, I don’t want to be marketed to when I’m trying to understand something unrelated. I have a business degree so I intimately understand that HN is technically free and it’s nice to get free eyeballs, but we are people too. I’m so much more than a credit card number, yet you’ve reduced me to a user acquisition in the most insulting way possible.

Perhaps instead of your ideas, it’s worth seeding your own personal make up with a firm statement of ethics??

Are you the kind of person who will hijack conversations to promote your product? Or do you have integrity?

Just purely out of concern for your business, do you have a cofounder who could handle marketing for you? If so, consider letting her have complete control over that function. It’s genuinely sad to see a founder squander goodwill on shitty marketing.

[nearting]

This doesn't address the question in any way except to note that you also use Merkle Trees. Do you reply to any comment mentioning TypeScript with a link to your Show HN post as well?

[gr4vityWall]

Sorry, but your post came off as blatant advertising. There is no need to link to your company announcement just because it benefits you.

[jchanimal]

Thanks y'all -- feedback taken. If I were saying it again I'd say something like:

Merkle proofs are rad b/c they build causal consistency into the protocol. But there are lots of ways to find agreement about the latest operation in distributed systems. I've built an engine using deterministic merge -- if anyone wants to help with lowest common ancestor algorithms it's all Apache/MIT.

While deterministic merge with an immutable storage medium is compelling, it doesn't solve the finality problem -- when is an offline peer too out-of-date to reconcile? This mirrors the transaction problem -- we all need to agree. This brings the question I'm curious about to the forefront: can a Merkle CRDT use a Calvin/Raft-like agreement protocol to provide strong finality guarantees and the ability to commit snapshots globally?

Apologies for the noise.

[csomar]

Crypto/Blockchain makes it harder to have an incorrect state. If you fk up, you need to take down the whole operation and reverse everything back to the block in question. This ensures that everything was accounted for. On the other hand, if you fk in a traditional ledger system you might be tempted to keep things running and resolve "only" the affected accounts.

[delfinom]

It's a question of business case. While ensuring you are always accounted correctly seems like a plus, if errors happen too often potentially due to volume, it makes more business sense sometimes to handle it while running rather than costing the business millions per minute having a pause.

[necovek]

It's mostly a different approach to "editing" a transaction.

With a blockchain, you simply go back, "fork", apply a fixed transaction, and replay all the rest. The difference is that you've got a ledger that's clearly a fork because of cryptographic signing.

With a traditional ledger, you fix the wrong transaction in place. You could also cryptographically sign them, and you could make those signatures depend on previous state, where you basically get two "blockchains".

Distributed trust mechanisms, usually used with crypto and blockchain, only matter when you want to keep the entire ledger public and decentralized (as in, allow untrusted parties to modify it).

[koolba]

> With a traditional ledger, you fix the wrong transaction in place.

No you don’t. You reverse out the old transaction by posting journal lines for the negation. And in the same transactions you include the proper booking of the balance movements.

You never edit old transactions. It’s always the addition of new transactions so you can go back and see what was corrected.

[necovek]

Right, thanks for the correction: I wanted to highlight the need for "replaying" all the other transactions with a blockchain.

[olddustytrail]

In git terms, it's like `revert` Vs `rebase`.

[lelanthran]

> With a traditional ledger, you fix the wrong transaction in place.

That's not how accounting works. You post a debit/credit note.

[solatic]

> With a blockchain, you simply go back, "fork", apply a fixed transaction, and replay all the rest.

You're handwaving away a LOT of complexity there. How are users supposed to trust that you only fixed the transaction at the point of fork, and didn't alter the other transactions in the replay?

[necovek]

My comment was made in a particular context. If you can go back, it's likely a centralized blockchain, and users are pretty much dependent on trusting you to run it fairly anyway.

With a proper distributed blockchain, forks survive only when there is enough trust between participating parties. And you avoid "editing" past transactions, bit instead add "corrective" transactions on top.

[im3w1l]

If its for internal why not just use a normal append only log. x amount transferred from account y to account z. A three column csv oughta do it.

[sneak]

Any time your proposal entails a “why not just”, it is almost certainly underestimating the mental abilities of the people and teams who implemented it.

A good option is “what would happen if we” instead of anything involving the word “just”.

[qazxcvbnmlp]

“Just” usually implies a lack of understanding of the problem space in question. If someone says “solution X” was considered because of these factors which lead to these tradeoffs however since then fundamental assumption Y has changed which allows this new solution then it’s very interesting.

[jknoepfler]

Sure. When I ask "why don't we just" I'm suggesting that the engineering solutions on the table sound over-engineered to the task, and I'm asking why we aren't opting for a straightforward, obvious, simple solution. Sometimes the answer is legitimate complexity. Equally as often, especially with less experienced engineers, the answer is that they started running with a shiny and didn't back up and say "why don't we just..." themselves.

[PittleyDunkin]

Counterfactuals strike me as even less useful than underestimating competency would be. Surely basic double-entry accounting (necessarily implying the use of ledgers) should be considered table stakes for fintech competency.

[foobarbecue]

Lots of threads on this here, most recently https://news.ycombinator.com/item?id=42038139#42038572 . I think this example is perfect, with the "oughta do it"

[tonyhart7]

it literally ledger, its only show where money went but not showing "why" the money move

double entry with strong check that ensure its always balance fix this

[foobarbecue]

https://news.ycombinator.com/item?id=42038139#42038572

[DanielHB]

> I hypothesized, given how shitty the codebase was, that they must be tracking this stuff poorly.

That is like half of the plot of Office Space

[mlloyd]

This sounds like a situation that I know about at the placed identified by name in your comment. It took months to track down the issue.