|
|
|
|
|
by jhoechtl
1691 days ago
|
|
> TLDR: Arrow and DuckDB provide fast and memory efficient database aggregates compared with R's RDS format and SQLite. Arrow and RDS were fast to load. I nowhere read that the authors come to the conclusion that SQlite does NOT provide fast and memory efficient database aggregates. Instead they conclude > SQLite and DuckDB files consists in a single large file (3 GB and 2.5 GB each), but the indexes we created allow their respective packages to read a copy of the tables that has just the year and reporter_iso columns, and therefore allows very fast filtering to provide the exact location of what we need to read in the large tables. |
|