Following the same logic, you should never buy a phone used in case the previous owner installed a rootkit, and may also want to worry about rogue factory workers installing a rootkit before packaging the phone.
I think running a system restore should make the phone reasonably safe?
Depending on your level of carefulness… I would consider your phone not secure if it’s confiscated and you’re not in the same room as it.