Please report this in https://zulip.com/development-community/; this certainly sounds like a memory leak. We aren't aware of any other reports of memory leaks in Zulip in the last couple years, so we'd be very happy to help track down what's happening on your server.
https://zulip.readthedocs.io/en/latest/production/requiremen... is a useful reference for folks interested in memory requirements for a self-hosted Zulip server.