No, it won't help to remove the SIM card. Unless you've removed the battery from your phone, then you can be tracked. And there's no such thing as "can't ditch my phone."
So, for instance, the FBI inputs selectors as the IMEI, not the phone number, when searching for people?
And there may be a legitimate reason why you may want to have a device. For instance, to run maps at a certain location. Or having data on the device you need once you arrive at a destination.
I worked telecoms a few years back. One of the things I worked on was an anti spam system for SMS. It had the ability to blacklist, throttle and log messages based on a number of identifiers including imei, phone number, Tower id, service center id and message content (typically binary patterns in non text payloads but there was nothing stopping it to be used to block messages containing certain text or keywords).
So if we were using these numbers to block messages then I'd absolutely expect government agencies to use them to monitor or track phones.
Would removing the SIM be at all helpful for avoiding tracking if you couldn't ditch a phone with an integrated battery?