That's not the whole story. Plastic is a chemical 'sponge' that soaks up contaminants from the environment (eg heavy metals, brominated flame retardants, pesticides, etc), concentrating them to many times background levels before delivering them into your body. For example household dust (which is >50% microplastics) is the primary route for brominated flame retardants to enter the human body.
And that's not even to mention intentional additives (phthalates, Bisphenol-*, BFRs again), which are far from inert.