Maybe. Not having a password on the server eliminates all the risks associated with weak or leaked passwords. And then you can configure SSH to reject password logins altogether. It's not an insignificant benefit.
I'd say there are more valuable things you can do to improve security than solving the problem of "having to ssh in with a password one time to upload a key, then updating the config to reject password logins".