https://lemire.me/blog/2012/05/31/data-alignment-for-speed-m...
TL;DR: 10% difference on what in 2012 was a low-end CPU, no difference on "new in 2012" CPUs. So my guess is that by now it really doesn't matter anymore :)