[tsimionescu]

There is nothing in the IPv4 header that is only useful for TCP, especially not in the parts removed from IPv6. Overall the IPv6 header gets rid of the 4 bytes of fields used for fragmentation, and 2 bytes used for the checksum. Fragmentation was never used for TCP in any sane implementation (as TCP can do fragmentation at the TCP layer), and the checksum were fully redundant for TCP. For UDP, the checksum used to be optional, but is now required. So, for an optimized implementation, the checksum removal isn't even a win for UDP, as it has just moved from the IP layer to the UDP layer.

So, we have added 24 bytes to the header because of the address difference, and removed 4 bytes from other places.

Now again, there are many differences between IPv4 and v6 that are much more relevant to latency than this extra header overhead. But it is a real overhead, there is no extra scope for payload. Your observation with ping is just wrong (most likely both versions are just padding the packets up to 64 bytes by default).