I really hope the rust implementation of Matrix helps. I reboot my Matrix container daily because if I don't it'll quickly eat through the 12GB of memory I've allocated it. I run a 3 person instance.
12GB of RAM sounds like something is badly wrong; a typical small server should use a few GB at most. Most likely is that either your users have joined loads of big rooms, or some malformed broken room, and the IO can’t keep up with all the traffic it’s seeing so things are stacking up in RAM. Please come to #synapse:matrix.org so we can help debug it.