So it seems this is their software defined network, the packets still look good when entering via the microsoft edge routers and they get bitflips on their way into our express route
after a full day of searching for a possible cause, someone from microsoft suggested that they try to move the express route gateways (which are seemingly VMs inside of Azure) to a different hypervisor. For now it seems like the first gateway was flipping bits and on new hardware it is running normal again... That was a very interesting day troubleshooting wise