A lot of people did and still do. And that was actually quite 'hard' because you had to touch micro electronics. If you make PCs (phones, laptops) out of Lego, people will do it much faster and more often. Going to a gig? Pull out the bluetooth, plug in the midi block. Going on the road in 2 countries? Pull an USB block, plug in a second 4G. Etc.
Edit; not saying this is a good idea for a phone, but to some extend it would be for a laptop (not ultrabook). Some rugged ones are quite like that.
Or buy the integrated phone that includes all of those features and doesn't make you swap them out and risk losing or damaging the blocks.
Modularity often ends up being a _cost_ to the unsophisticated majority of end users. People don't like it in their software (preferring a "complete solution"), nor in their hardware (e.g. cheap integrated hifi versus expensive minority separates).
Desktops are still made from interchangeable components.
I upgraded RAM, video card, sound card, added more hard drives, retired old ones. Later on I replaced CPU+motherboard (unfortunately you can rarely just upgrade the CPU).
If you're not lazy, this can save you quite a lot of money.
Phones are nothing like PCs, and will not be for a long time. It would take some crazy engineering to make them lego-like, or even plain upgradeable.
Edit; not saying this is a good idea for a phone, but to some extend it would be for a laptop (not ultrabook). Some rugged ones are quite like that.