Granted, there is also growth of dust grains. But surfaces from which atoms can be ejected by UV photons will erode.
Some notes: https://www.astro.princeton.edu/~draine/dust/Draine_IPMU_Lec...