[staticassertion]

This is already a problem if you're removing an item, regardless of whether it's done with swap remove or not.

[toast0]

For the item you're removing, you are presumably removing it because all handles have been closed. But the item you're potentially swapping with may certainly have open handles.

[staticassertion]

True.