It's not the pointers themselves so much as what they're typically used for. How would you do dynamic sizing? Imagine sending just a struct of integer arrays this way, you'd have to either know their sizes ahead of time or just be ok with sending a lot of empty bits up to some max size. And recursive structures would be impossible.
You could get around this with a ton of effort around serdes, but it'd amount to reinventing ASN1 or Protobuf.
I can see that in niche situations, particularly if you have a flat structure and uniform hardware. Cap'n Proto is also a way to do zero-parsing, but it has other costs.
You could get around this with a ton of effort around serdes, but it'd amount to reinventing ASN1 or Protobuf.