[ben_w]

Micro-meteors; frictional wear and tear from normal use; proton ablation from the solar wind[0].

I'd assume we couldn't even get to that scale without solving vandalism, war, and insanity, but if not, then over the scale of a million years there will be twenty thousand space-Victorians and space-Taliban having space-Jihads against space-Buddha- and space-Baphomet-statues. I dread to think what the K2 version of the deliberate destruction and death of WW1 and WW2 would be like.

Likewise industrial accidents (space Chernobyl?), but if the big ones aren't solved there's a significant chance of a Kessler cascade rather than just, say, small incidents destroying 1% of the habitats every millennia[1].

[0] it doesn't get very deep on geological scales, but Dyson swarms aren't capable of being very deep on geological scales either.

[1] Completely arbitrary percentage of course, but that percentage would destroy half of what remained every 69-ish millennia.