It's particularly tricky when dealing with async code, where you can't simply say "block here till you have 4 bytes." If you're just getting in events that say "you received n bytes and here they are."
All networking code should work even if it receives 1 byte at a time. Use a buffer, and have some sort of abstraction responsible for packetizing the input. The output of that module is a fully formed frame ready for interpretation.