This is a good point. I think a simple remedy would be to include the IP the server should allow connections from as part of the authenticated payload from the client in the request.
That defeats the purpose of port knocking. If you know which IP connections comes from and you trust it - just allow connections in firewall. Port knocking is for temporary allowing certain incoming IPs.