Well, after changing the ssh port to something really big, OOM and heavy CPU usage stopped, as I was still using that public IP, so concluded it was not an inside job .
There were like thousands of requests in an hour, and that went on continuously, before I changed the port.
There were like thousands of requests in an hour, and that went on continuously, before I changed the port.