Didymos has an orbit that only very occasionally brings it anywhere near Earth. Only within a few million kilometres every hundred years or so. The small amount of debris produced will continue along the slightly altered orbit, so won't be a hazard to local space traffic until such time as the larger asteroid comes close enough to hit Eartch, at which point the tiny debris is a lesser concern.
And the solar system is anyway full of flying gravel. The environmental impact is similar if you went to an open mine pit and hammered at a small rock for a while.
I think the concern is that near earth orbit one day will be impassable without some garbage collection.
The acceleration of objects in near earth orbit is vastly greater than the particles from a mine. Space ships collaring with super fast particles in the orbit could spell disaster for the spacecraft.
Hopefully we find ways to make the rock impact negligible if we ever get to the stage of unable to leave orbit due to collisions with space particles (Not saying subatomic particles I just mean debris)