That's only true for certain movements, performed for a certain period of time and potentially with only a cetain amount of force applied to your body. A movement performed incorrectly, performed for too long or with too much force will damage your body.
And of course, this only applies, if the stress doesn't take the form of your head being hit by a hammer.
Besides whether your mind is physical or not doesn't really matter all that much. What matters is that it's mutable. It can be changed, for better or worse and that change depends on the environment.
And if you think about it, stressing your body makes it stronger. That's why working out builds muscles.