Your face is only authenticating you to your device because that's what you chose. If you don't want that (e.g. your identical twin sister loves pranking you) you can just use a different authenticator. The remote web site deliberately has no idea your face was involved, it just knows your identity was verified on its behalf by the hardware storing your private key.
I can have a near infinite number of passwords, but I only have 1 face, or 10 fingers. When all of my fingerprints are compromised, and the system only allows fingerprint login, now what do I do ?
Your face is only authenticating you to your device because that's what you chose. If you don't want that (e.g. your identical twin sister loves pranking you) you can just use a different authenticator. The remote web site deliberately has no idea your face was involved, it just knows your identity was verified on its behalf by the hardware storing your private key.