[dionian]

Have you considered something like iceberg tables?

[aynyc]

Yes, but parquet hates small files.

[dionian]

You can't compact? i.e. iceberg maintenance

[aynyc]

We might be doing something wrong, but we saw significant performance degradation for both ingestion and query when doing compaction when it comes to finance data during trading hours.