[xtracto]

This may be useful for somebody: We are also using DuckDB heavily at my workplace (we do Tax analytics of very large companies with huge amounts of data). We have certain DuckDB processes that happened in AWS infrastructure, where the data is saved in GP3 disks.

We didn't know that for GP3 disks, you can increase not only IOPS but also Read/Write Throughput [1] which by default is 125 MB/s. So by default we were not seeing the performance we expected.

Once we increased the throughput of the EBS, it was amazing. So if you are not seeing the performance you read about online when using DuckDB, it may be something like that.

[1] https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-p...

[pants2]

This seems crazy low to me. AWS has default 3K IOPS and 125 MB/s throughput, meanwhile my Macbook Pro has 700K IOPS and 14.5GB/s throughput.

Is Amazon running on super outdated legacy networking?

[kristjansson]

SAN vs local. Local NVME (“instance storage“) on AWS is wicked fast too, but live and dies with the instance

[_zoltan_]

in 2026 you can full the PCIe BW from remote storage at roughly the same latency as your local NVMe drive (hello NVMeoF).

your comment would have been valid 10 years ago, maybe.

[tjoff]

Well, cloud providers aren't exactly known for performance so the comment is still valid and will be for quite some time still.

[SOLAR_FIELDS]

Mind unwrapping this a bit more? I’m interested in iops heavy distributed caching and this seems like it unlocks some creative use cases there