[belmarca]

Nice. The book contains a bunch of good information that could already be found elsewhere but collecting it is quite practical. I highly suggest to read Kleppmann's Designing Data-Intensive Applications. The first edition was very good, a second one came out recently.

I was CTO of a FinTech where I built the whole software stack from scratch: the lessons in the book are mostly correct. I say mostly, because as always, there is a lot of "it depends" to take into consideration for your particular project. For example, I chose to not use event-sourcing to avoid the whole state computation issue. A standard append-only audit trail can do the job.

You can't guarantee exactly-once delivery but you can construct effectively-once processing, and that is what you really want.

Store every request and response : absolutely, and not only when consuming APIs, but when collecting any information from the outside world (and, if you can, also log every intermediate transformation step within your perimeter). Content-adressed buckets + a relational table are great for this.

The text also does not mention anything about data lineage. What happens if a vendor updates some data mid-day that you absolutely need to be aware of? You need to be able to account for that, while also re-playing computations that used the old values and get the same result. It's not a particularly hard problem to solve, but it takes some thought.

[potamic]

Why content-addressed storage? I'm assuming this is mostly for auditability after the fact? Wouldn't it make sense to use some business identifier to be able to refer it back easily?

[belmarca]

You will need a combination of the two. The filename should contain a compound business identifier, essentially a tuple of appropriate facts, as well as a (truncated) hash of the content.

And yes, the reason is auditability but also bare necessity.

Imagine vendor A publishes fresh files every day, but you don't exactly know when. And suppose that the vendor can (and does) republish some of them at any time, with or without notice. You need to watch those files for changes and update your own sources accordingly.

For the collection and dedup, you can construct a tuple of meta-information that will give you a good idea if the file is the same (name, time last modified, file size, etc). Then only if you're still unsure do you download the file and hash it, compare with what you already have, and make a decision. (What if the file name changed slightly and now you must re-categorize the data in the file, or what if a single row was added or removed, etc).

It's also useful when auditing (tamper-proof) or when re-running or debugging calculations that used the data in question. If a user used the data available at 9AM, which was unfortunately incomplete and led to issues, having a full data lineage really helps. You can trivially re-run the computation with the information available at that time to see what happened. Or you can mark some results as stale, etc.

There are different ways to do this and one is keeping track of your content-adressed files in a relational table and give each row a unique identifier, so that you can tell that this particular row in your AAPL OHLCV data comes from this particular file which was fetched at this particular time etc, etc. Pair that ID with a timestamp and when doing computations you can query for "the latest" AAPL data while storing the exact file ID (time is fuzzy, content hash is not). Look into temporal tables and the like.