[ryao]

The performance degradation is likely from full meta slabs and maybe from gang blocks, although ZFS does a fair job at preventing gang blocks by using best fit behavior to minimize the external fragmentation that necessitates them. The magic threshold for best fit behavior is 96% full at the meta slab level. This tends to be where slowdowns occur. On spinning disks, being near full also means that basically all of the outermost tracks have been used, so you are limited to the inner most tracks, which can halve bandwidth.

Anyway, it would be nice if you could provide actual numbers and meta slab statistics from zdb. The worst case fragmentation that has been reported and that I can confirm from data provided to me is a factor of 2 reduction in sequential read bandwidth on a pool consisting of spinning disk after it had reached ~90% capacity. All files on it had been created out of sequence by bit torrent.

A factor of 2 might be horrible to some people. I can certainly imagine a filesystem performing many times worse though. I would be interested to hear from someone who managed to do worse than that in a manner that cannot be prescribed to the best fit allocator protecting the pool from gang block formation.