Why not block all ip's that don't belong to your isp since you're the only one who'd need to ssh to your server? There are some good sources out there to find out your ip range
I was wondering if that was possible. But, I don't think the range of ips would be accurate, since they might have multiple gateways(?). Also, if I'm traveling, I'd need a home VPN to whitelist.