[dpcan]

What happens when you start to accumulate over 5 million+ records (as an arbitrary large number)? Or a couple gigs of stats data?

We used to provide our own stats software to our customers before it became an utter nightmare of indexing databases, cron jobs, report generation, and customers stepping over their hosting quotas because of all the stats data.

[rkwz]

If your reports are predefined - summarised and no adhoc reporting needed, you can look into the continuous view feature of PipelineDB - http://docs.pipelinedb.com/continuous-views.html

> As soon as a stream row has been read by the continuous views that must read it, it is discarded. Raw, granular data is not stored anywhere. The only data that is persisted for a continuous view is whatever is returned by running a SELECT * FROM that_view. Thus you can think of a continuous view as a very high-throughput, realtime materialized view.

I use PipelineDB continuous views in Freshlytics - https://github.com/sheshbabu/freshlytics/blob/master/src/ser...

TimescaleDB also has something called "Continuous Aggregates" - https://docs.timescale.com/latest/using-timescaledb/continuo... but I haven't looked into it much.

[cosmie]

I can't speak to the OP's performance, but Snowplow[1] may be good for your use. It's a bit more general purpose than just pure web analytics, and doesn't come with the reporting interface. But as far as the data collection and storage at scale aspect, it's fantastic.

[1] https://snowplowanalytics.com/products/snowplow-open-source/

[electerious]

> What happens when you start to accumulate over 5 million+ records

I don't think Ackee is the right tool if your site as so many visitors. And to be honest: I don't know what will happen at this scale. There shouldn't be a problem as long as MongoDB can handle it.

[nodesocket]

Agree, just make sure you are running a modern version of MongoDB with WiredTiger storage engine and the server has enough memory. It should be able to handle 5 million records on modest hardware.