Every kilogram of mass added to the return trip increases the required deceleration, fuel, etc. Not cheap. At some point you can't reach orbit with the required fuel to get everything back. The Falcon 9 boosters are nearly empty when they come back. As a matter of fact, at least one of their early landing failures was because they cut the fuel load too close and ran out of fuel just before reaching the ground. Minimizing weight on the return landing is critical.