[jgrahamc]

A lot of it is log data from requests passing through CloudFlare. We run them through Kafka and consumers do stuff like attack detection and generate statistics for our customers. We have about 4 million sites on CloudFlare and each customer has access to analytics about their site which are stored in a CitusDB database.

[paulasmuth]

Impressive. Any chance you could tell how much data is stored for the analytics service after pre-aggregation? (In terms of TB/day or so - I guess it can't be the full 70gbit/s?).

[jgrahamc]

No, nothing like that much data. It's aggregates like "requests per second" and "attacks per second" etc. The actual log lines aren't stored.

More details: https://blog.cloudflare.com/scaling-out-postgresql-for-cloud...

[paulasmuth]

Thanks. I had read that article some time back :)

Sadly it does not give a figure/order of magnitude of the amount of data that's stored in citus after aggregation, but I guess it's just not public information. [I'm working on a system that is somewhat similar to CitusDB (eventql.io) and am always really interested in these numbers]

EDIT: I can't reply to your other comment for some reason but many thanks for digging that up, it's very interesting info to me!

[jgrahamc]

I'd be happy to make the numbers public I just don't have them in front of me. It looks like data is going into the CitusDB cluster at about 15Mbps.