[Dylan16807]

Reading each byte once only takes a few microseconds. That's not the issue.

What you need is for the slow core of the algorithm to be fixed-speed.

Either by only reading the input bytes during initialization, or by only feeding a fixed number of input bytes into the core during each round.