|
|
|
|
|
by ed25519FUUU
2031 days ago
|
|
Side note, they really should have in big-bold letters "DO NOT ENABLE DEDUPLICATION UNLESS YOU HAVE A TON OF RAM!" on their readme. That was a huge mistake on my part. The ram requirements are VERY high for good performance. I realized how bad the performance was when it took about 2 hours to delete 1000 files. |
|
Deduplication is the process for removing redundant data at the block level, reducing the total amount of data stored. If a file system has the dedup property enabled, duplicate data blocks are removed synchronously. The result is that only unique data is stored and common components are shared among files.
Deduplicating data is a very resource-intensive operation. It is generally recommended that you have at least 1.25 GiB of RAM per 1 TiB of storage when you enable deduplication. Calculating the exact requirement depends heavily on the type of data stored in the pool.
Enabling deduplication on an improperly-designed system can result in performance issues (slow IO and administrative operations). It can potentially lead to problems importing a pool due to memory exhaustion. Deduplication can consume significant processing power (CPU) and memory as well as generate additional disk IO.