Swapping on the SD card is a nightmare. I found out that disabling the swap actually leads to a better experience, even if it means that firefox gets killed from time to time.
Would swap on zram work? I've used that to help with low-memory situations, but I'm not sure if the Pi's CPU is fast enough to avoid it being a different bottleneck.