| HN Mirror

rosetremiere 1208 days ago

The faq helped, thanks! So, an example of typical use would be, say, as the internal ledger for a company like (transfer)wise, with lots of money moving around between accounts? But I understand it's meant to be used internally to an entity, with all nodes in your system trusted, and not as a mean to deal with transactions from one party to another, right?

https://docs.tigerbeetle.com/FAQ#what-is-two-phase-commit

eatonphil 1208 days ago

Yes that's a good example!

And you can model external accounts that have their own confirmation process using our two-phase transfer support.

Great to hear! Joran from TigerBeetle here.

Yes, exactly. You can think of TigerBeetle as your internal ledger database, where perhaps in the past you might have had to DIY your own ledger with 10 KLOC around SQL.

And to add to what Phil said, you can also use TigerBeetle to track transactions with other parties, since we validate all user data in the transaction—there are only a handful of fields when it comes to double-entry and two-phase transfers between entities running different tech stacks.

The TigerBeetle account/transfer format is meant to be simple to parse, and if you can find user data that would break our state machine, then it's a bug.

Happy to answer more questions!

ngrilly 1208 days ago

It's a distributed database for financial transactions, using double entry accounting, written in Zig, and with a very innovative design:

- LMAX inspired

- Static memory allocation

- Zero copy with Direct I/O

- Zero syscalls with io_uring

- Zero deserialization

- Storage fault tolerance

- Viewstamped Replication consensus protocol

- Flexible Quorums

- Deterministic simulation like FoundationDB

Zero deserialization? That sounds rather scary. This means absolute trust in data read from disk or received from other nodes?

rom-antics 1208 days ago

What is the threat model you're worried about? If an attacker can write data to your disk or authenticate to your cluster, aren't you already screwed?

Yes, these are exactly my threats.

First, because I'm a strong believer in defense-in-depth. Secondly because both disk corruption and network packet corruption happen. Alarmingly often, in fact, if you're operating at large scale.

Ours too!

For example, our deterministic simulation testing does storage fault corruption up to the theoretical limit of f according to our consensus protocol.

Details in our other reply to you.

Great question! Joran from TigerBeetle here.

  "This means absolute trust in data read from disk or received from other nodes?"

TigerBeetle places zero trust in data read from the disk or network. In fact, we're a little more paranoid here than most.

For example, where most databases will have a network fault model, TigerBeetle also has a storage fault model (https://github.com/tigerbeetledb/tigerbeetle/blob/main/docs/...).

This means that we fully expect the disk to be what we call “near-Byzantine”, i.e. to cause bitrot, or to misdirect or silently ignore read/write I/O, or to simply have faulty hardware or firmware.

Where Jepsen will break most databases with network fault injection, we test TigerBeetle with high levels of storage faults on the read/write path, probably beyond what most systems, or write ahead log designs, or even consensus protocols such as RAFT (cf. “Protocol-Aware Recovery for Consensus-Based Storage” and its analysis of LogCabin), can handle.

For example, most implementations of RAFT and Paxos can fail badly if your disk loses a prepare, because then the stable storage guarantees, that the proofs for these protocols assume, is undermined. Instead, TigerBeetle runs Viewstamped Replication, along with UW-Madison's CTRL protocol (Corruption-Tolerant Replication) and we test our consensus protocol's correctness in the face of unreliable stable storage, using deterministic simulation testing (ala FoundationDB).

Finally, in terms of network fault model, we do end-to-end cryptographic checksumming, because we don't trust TCP checksums with their limited guarantees.

So this is all at the physical storage and network layers.

  "Zero deserialization? That sounds rather scary."

At the wire protocol layer, we:

  * assume a non-Byzantine fault model (that consensus nodes are not malicious),
  * run with runtime bounds-checking (and checked arithmetic!) enabled as a fail-safe, plus
  * protocol-level checks to ignore invalid data, and
  * we only work with fixed-size structs.

At the application layer, we:

  * have a simple data model (account and transfer structs),
  * validate all fields for semantic errors so that we don't process bad data,
  * for example, here's how we validate transfers between accounts: https://github.com/tigerbeetledb/tigerbeetle/blob/d2bd4a6fc240aefe046251382102b9b4f5384b05/src/state_machine.zig#L867-L952.

No matter the deserialization format you use, you always need to validate user data.

In our experience, zero-deserialization using fixed-size structs the way we do in TigerBeetle, is simpler than variable length formats, which can be more complicated (imagine a JSON codec), if not more scary.

> Where Jepsen will break most databases with network fault injection, we test TigerBeetle with high levels of storage faults on the read/write path, probably beyond what most systems, or write ahead log designs, or even consensus protocols such as RAFT (cf. “Protocol-Aware Recovery for Consensus-Based Storage” and its analysis of LogCabin), can handle.

Oh, nice one. Whenever I speak with people who work on "high reliability" code, they seldom even use fuzz-testing or chaos-testing, which is... well, unsatisfying.

Also, what do you mean by "storage fault"? Is this simulating/injecting silent data corruption or simulating/injecting an error code when writing the data to disk?

> validate all fields for semantic errors so that we don't process bad data,

Ahah, so no deserialization doesn't mean no validation. Gotcha!

> In our experience, zero-deserialization using fixed-size structs the way we do in TigerBeetle, is simpler than variable length formats, which can be more complicated (imagine a JSON codec), if not more scary.

That makes sense, thanks. And yeah, JSON has lots of warts.

Not sure what you mean by variable length. Are you speaking of JSON-style "I have no idea how much data I'll need to read before I can start parsing it" or entropy coding-style "look ma, I'm somehow encoding 17 bits on 3.68 bits"?

> Also, what do you mean by "storage fault"? Is this simulating/injecting silent data corruption or simulating/injecting an error code when writing the data to disk?

Exactly! We focus more on bitrot/misdirection in our simulation testing. We use Antithesis' simulation testing for the latter. We've also tried to design I/O syscall errors away where possible. For example, using O_DSYNC instead of fsync(), so that we can tie errors to I/Os.

> Ahah, so no deserialization doesn't mean no validation. Gotcha!

Well said—they're orthogonal.

> Not sure what you mean by variable length. Are you speaking of JSON-style "I have no idea how much data I'll need to read before I can start parsing it"

Yes, and also where this is internal to the data structure being read, e.g. both variable-length message bodies and variable-length fields.

There's also perhaps an interesting example of how variable-length message bodies can go wrong actually, that we give in the design decisions for our wire protocol, and why we have two checksums, one over the header, and another over the body (instead of one checksum over both!): https://github.com/tigerbeetledb/tigerbeetle/blob/main/docs/...

Alright, I'm officially convinced that you've thought this out!

So, how's the experience of implementing this in Zig?

ngrilly 1208 days ago

I can't stop being amazed by TigerBeetle's design and engineering.

Thank you Nicolas!

laserbeam 1208 days ago

It's a special purpose DB. No relation to blockchains.