10+ years ago I worked on a trading system that was generating something like 1TB/day of messaging.
As we hit these levels we asked them - how many trades are we even doing on this system? The answer was something on the order of.. 50. Granted it was a bond system and the nationals are huge, but theres just no reason to store 20GB per trade.
These are the kinds of decisions that get made when one team is responsible for message generation and the other is responsible for the storage, lol.
We then had to work backwards with them to unwind a lot of the INFO level chatty messaging between what you'd now call "microservices" and reduce the volume by 90+%.
I suppose you need to know how many requests did not result in a purchase. Is it 1000 views:purchase? I have not checked in on a Prime Day sale for several years, but is there any timeliness component (Flash Sales?) where people would be incentivized to mash the reload button?
After looking at screen after screen of no-name garbage on Prime Day, I gave up. I suspect that there are tons of people like me. In other words, we only contributed to the numerator, not the denominator.
As we hit these levels we asked them - how many trades are we even doing on this system? The answer was something on the order of.. 50. Granted it was a bond system and the nationals are huge, but theres just no reason to store 20GB per trade.
These are the kinds of decisions that get made when one team is responsible for message generation and the other is responsible for the storage, lol.
We then had to work backwards with them to unwind a lot of the INFO level chatty messaging between what you'd now call "microservices" and reduce the volume by 90+%.