Two likely reasons. First, the previous form factor didn't fit in all TVs, so they had to include a tiny HDMI extender cable; this form factor has built-in tolerance for tight fits. Second, the new form factor allows more room for hardware bits and heat dissipation.
Also probably because some television sets can block wifi signals up close. I had a weak wifi connection on mine and a 3 ft hdmi extender fixed it (which was probably overkill).
I had this as well. Took me a while to figure out it was my Chromecast. When I watch TV I basically have to turn of my Chromecast because of the otherwise interference it causes.
I just installed a couple old CCs at the office, and the old rigid form factor can sometimes be tricky to plug into some TVs, due to the mounting brackets sitting right on top of the video connectors. Bad design from the TV makers? Yes, but the flexible dongle would never have this issue.
Probably the fact that you couldn't plug it into a lot of TVs without using an HDMI extender cable (due to how ports are laid out). That's why the HDMI connector is now on a flexible cable.