The problem I see here is the more senses you engage - really, the more coordination you demand of the user, the harder it is to make the interface feel intuitive.