[axegon_]

I use duckdb HEAVILY at work and it's been a game changer. I'm sifting through terabytes of data multiple times a day, mixing, matching, updating, filtering, DuckDB is second to none. For anyone that hasn't used it: you are missing out.

[xtracto]

This may be useful for somebody: We are also using DuckDB heavily at my workplace (we do Tax analytics of very large companies with huge amounts of data). We have certain DuckDB processes that happened in AWS infrastructure, where the data is saved in GP3 disks.

We didn't know that for GP3 disks, you can increase not only IOPS but also Read/Write Throughput [1] which by default is 125 MB/s. So by default we were not seeing the performance we expected.

Once we increased the throughput of the EBS, it was amazing. So if you are not seeing the performance you read about online when using DuckDB, it may be something like that.

[1] https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-p...

[pants2]

This seems crazy low to me. AWS has default 3K IOPS and 125 MB/s throughput, meanwhile my Macbook Pro has 700K IOPS and 14.5GB/s throughput.

Is Amazon running on super outdated legacy networking?

[kristjansson]

SAN vs local. Local NVME (“instance storage“) on AWS is wicked fast too, but live and dies with the instance

[_zoltan_]

in 2026 you can full the PCIe BW from remote storage at roughly the same latency as your local NVMe drive (hello NVMeoF).

your comment would have been valid 10 years ago, maybe.

[tjoff]

Well, cloud providers aren't exactly known for performance so the comment is still valid and will be for quite some time still.

[SOLAR_FIELDS]

Mind unwrapping this a bit more? I’m interested in iops heavy distributed caching and this seems like it unlocks some creative use cases there

[moribvndvs]

I recently got a task to sift through a daily drop of parquet files that need to be processed and given a UI for arbitrary queries by a non-technical team. Naively I started by looking at redshift, Athena/glue, etc. I spun my wheels for a week and barely made progress, resulting in a miserably fragile, slow, overly complicated, and/or expensive set of underwhelming PoCs. So I decided to give DuckDB a shot and I got the whole project done in under a day. It was so intuitive and runs so fast I was sure I was missing something. I wasn’t. What an absolute joy to use, and the supporting documentation is outstanding. My problem now is it’s my favorite hammer and I want everything to be a nail.

[aamargulies]

I asked Claude which tools it would most want to add to a machine we were working on. #1 was DuckDB.

[Jgrubb]

I asked Claude some ways I could make my teeth whiter, it suggested DuckDB!