I went to French immersion in Canada. We didn't 'miss out' on anything that kids did in English school. They still had to take a second language class (we didn't since we already had English + French). We just had classes like science, math and social studies in French as well. We still learned English, and of course live in an English-speaking region, so we missed out on nothing.
You miss out on special education support. Have reading troubles? Behind the curve in math ability? Have motor skills issues that make writing difficult? You're told to drop out of French immersion and return to the main English stream. It's a de-facto removal of anyone with any academic problems at all.
And by extension, English schools that overlap with French immersion schools become biased towards kids with learning difficulties which makes them that much less attractive to high-achieving parents. The TDSB elementary system isn't supposed to be streamed but effectively it is.
(I had two kids in TBSB French immersion until grade 4 when we moved away)