I think it's slightly more nuanced than that. It's more like "the walled garden is bound to fail and allow malicious apps so you might as well allow people to load whatever apps they like".
The argument is rather that the AppStore is not secure at all, but a) we still cannot sideload apps and b) legitimate apps get kicked off for no good reason, etc.
The problem is false security and false expectations.
Arguably it's easier for a user today to accidentally download a scammy app from the app store than it would ever be to do when sideloading. The latter would always come with the expectation and warning that it's not vetted by anyone, so the user and/or community will invest the time to make sure the download is legitimate.