I don't understand why was the next version of IP not just identical to IPv4 but with more bits in address space? Were they trying to do too many things at once in the 90's?
I don't think it not being that harms it as much as people think. It has to require updates for everything either way, people by and large don't care about "oh but it's only a small total breakage, going to jump on that then". On the other hand, yes, there certainly was some "we break everything anyways, so lets 'improve' things", combined with those improvements being designed at the wrong time, with assumptions that not always turned out to match reality. (E.g. a bunch of pieces that were added to IPv6 kind of assumed that routers would stay as they were, with routing done on CPUs, in software. Which they obviously didn't, and specialized hardware works on entirely different constraints)
It was fixing (or trying to) issues with the v4 spec that were now very apparent.
For example, ipv4 technically has a link-local address space but barely anything will use it and even less will successfully. Many other 80/90s protocols did much better at that (IPX being an example) as well as having distributed name and service locators and such.
IPv6 local networks of IoT devices or whatever can pretty much automagically start communicating with zero configuration to anything else locally. No DHCP or whatever required.
The world didn't stand still between v4 and v6, it'd be weird if the protocol did.
There still hasn't been a coherent answer. The best approximation is that a lot of networking researchers saw their once-in-a-lifetime opportunity to bikeshed the lowest layer of the public network stack, and couldn't resist the urge.
All of this could have been avoided by making NAT64 part of the original IPv6 standard (instead of wasting a decade pretending like it wasn't necessary) and making it a mandatory service provided by every IPv6 router, unless all downstream routers already provide the service. This would have forced an invisible-to-endpoints IPv6 transition, starting at the backbone ("no default route" region). Carriers would have very quickly pushed the NAT requirement downstream (away from the default-free region) in order to offload the burden it creates. Each step of the transition would have been a strictly local change between two peers, one of which is paying the other, and can therefore can be incentivized ("hey, if you run NAT64 for your downstreams so we don't have to, we will charge you less per month"). No global coordination, and the local coordination is always across a commercial relationship where a carrot can be dangled.
But instead we got what is basically a flag day, so "the transition" will never ever happen. We'll still be talking about it in twenty more years.
IPv6 is pretty much IPv4 with more bits - at least compared to the alternative of the time that was CLNP/DECNet (which had 20byte addresses and is quite different).
And the very slow transition hasn't really anything to do with the standard itself, but with the transition technologies (NAT64 like you mentioned).
It would have happened with any of the proposed alternatives. Hindsight and all..
NAT itself wasn't even a thing yet (not an RFC anyway) when IPv6 was being developed.
Flag days had worked in the past too, so why not again?