Lighter materials that would provide the same level of durability, security, insulation, and fire resistance would be significantly more expensive. No one wants to pay for that.
The other three walls don't have the same demands put on them. For one, you wouldn't need to lift them by hand, so they can be practically as heavy as you want.