If you are shifting correctly that is not the case. The "breaking" is achieved by the engine, the wear on the transmission is the same as when you are stepping on the gas.
It also cause pressure waves in the exhaust system since you're using the vacuum forming in the engine to slow you down, which is more noticable at low rpms due to the slowing of the positive exhaust pressure changing to intake vacuum frequency. Going from acceleration to engine braking at low rpms, such as in traffic jams, has caused a weaken seam in my exhaust system to break.