You can; but I'd not be surprised if there are commercial setups that use IPv6 packets for everything, including situations with IPv4 addresses.
I think it would allow getting rid of most of the ethernet header overhead, by basically just talking bare IPv6 packets.