[nelop]

The problems was all my fault :-) I was trying to use a port that was not designed for serial data. When data was sent across it was getting mashed. I think it's because both ports were not uart, therefore when the binary data was sent if they were not perfectly in sync it would get mashed, I might have been able to solve it by sending a clock as well. But the easier option was to just change everything to the uart ports and it magically worked.

[StevenThompson]

It was probably due to the lack of flow control. Serial doesn't work well when it's one-way. I did something similar to send logs waaaay back in the day, and it would constantly flip bits or send characters out of sequence, etc. I had to transmit very slowly to get it to work stably without any flow control. I want to say that I limited it to 9600 kbps before it started to become reliable.