You want your secret to have a secret? Several apps that provide time based one time password storage/management can also be secured with bio-metrics, pins or passwords (Aegis, MS Authenticator). Are you specifically thinking physical tokens only?
What do you mean by that? The TOTP standard doesn't specify how (if at all) the client is secured. Besides, the one-time code is used in addition to a password, not as a substitute for one.