Exactly. You can have a rock-solid password login system, but pretty much everyone has a reset feature which kicks a link to the email address, and allows that user to change the password. It's no less secure than that.
It is slightly less secure than that as the password reset form has a notification mechanism built into it in that the next time you login you realise that you need to reset the password back to what you control, where the magic link does not have such a mechanism to let you know that a compromise has taken place. It is slight, and requires you to be paying attention.
I don't think it's that slight a difference - password reset is akin to a door that you have to break open. The reset is obvious, there's evidence.
A password-less login, the email can be deleted and unless there are logs of last login, and the user notes the pertinent detail, then the intrusion can be covert.
Having someone unknowingly have a key to your apartment is much more of a breech than discovering a broken lock that you know needs fixing.