Hacker News new | ask | show | jobs
by seektable 436 days ago
Common approach is to use a data warehouse that is suitable for executing OLAP queries (to load/calculate data for your dashboards) in seconds. In simplest cases ('small data') this can be simply app's database replica - a separate (read-only) DB server that is used for reporting/analytics purposes. It is easy to configure master-slave replication with built-in SQL Server/PostgreSql/Mysql capabilities.

If replica server cannot execute queries fast enough, a specialized (optimized for OLAP-workload) database should be used instead. This can be a cloud service (like BigQuery, Redshift, Snowflake, Motherduck) or self-hosted solutions (ClickHouse, PostgreSql with pg_analytics extension, or even in-process DuckDB). Data sync is performed either with scheduled full-copy (simple, but not suitable for near real-time analytics) or via CDC (see Airbyte).