Browsers open pop-ups to ask "Can I run that application?" but only if that application is installed. If that application is not installed, the browser will ignore the custom URL.
It looks like a mitigation might be that in the event you do not have the application installed, to return a "denied" status and send a prompt to the user like "Unknown application protocol".
Something like that could still would be susceptible to a timing attack though.
Yes I believe the proper fix would be to always behave as if a popup is showing, independent of weather or not it actually shows.
Through it's maybe slightly more complex as you might need to behave as if the user clicked cancel in a way where a attacker can not easily differentiate it from an actual user clicking cancel.
Something like that could still would be susceptible to a timing attack though.