In the blog article[1] that linked to, it says "Unified transactional and analytical workloads: Lakebase integrates seamlessly with the Lakehouse, sharing the same storage layer across OLTP and OLAP. This makes it possible to run real-time analytics, machine learning, and AI-driven optimization directly on transactional data without moving or duplicating it."
Is the "without moving or duplicating" part actually a true statement? If the actual table state is only reconstructed by the pageserver, its not like Spark can just read it from S3.
Read replicas can be "shallow". You don't need to replicate all the data to create a replica. This allows to create them very very quickly (sub second).
All the extension still work. We don't support Citus today, but mostly because customers are not asking for it rather due to technical limitations. We support lots of extensions: https://docs.databricks.com/aws/en/oltp/projects/extensions
Thanks for offering. In the graph labeled "Prod customer throughput: (higher is better)" eyeballing it within a week you are seeing ~2k qps peak increase over the previous week.
Operationally, how do you handle landing that large of a perf improvement? If my data store changed that much in a week it could break something.
Generally the more throughput the system supports the better. In this case we were hitting limits (btw each operation is many queries of different sizes) and the customer observed higher latencies which is typical if the system can't sustain the throughput required.
After this change latencies are back to normal and throughput increased.
* Rate limiting on proxy in front of compute fleet
* Large tenants are broken up into shards, reducing hotspots
* Each shard is throttled to a fixed req/s rate
* We do not run pageservers at their redline in terms of CPU load, so there is some slack to take up bursts
* Capacity quotas which selectively throttle write traffic to the largest databases if they are competing with others for disk space, until the larger database is migrated away.
Is the "without moving or duplicating" part actually a true statement? If the actual table state is only reconstructed by the pageserver, its not like Spark can just read it from S3.
[1] https://www.databricks.com/blog/what-is-a-lakebase