Also keep in mind that the Raspberry Pi Zero (used by the person you're responding to) has the same processor as the original Pi. I'm certain that the Pi 4 does tremendously better at handling crypto than the original Pi processor.
The 64-bit versus 32-bit thing is almost certainly software wonkiness, as that forum post indicates. There's no reason the 64-bit mode should be slower than 32-bit mode for that task, and benchmarks often show that 64-bit ARM is actually measurably faster than running in 32-bit mode, due to the additional available registers and whatnot.
Are you using RSA or ECDSA certificates? RSA signing is a lot more difficult than ECDSA signing. If you're concerned about compatability with older clients, you can have dual certificates, but that can be more time consuming to setup.
https://www.raspberrypi.org/forums/viewtopic.php?p=1705614#p...