How common is this in Linux development? Is it better to have two competing solutions and "may the best one win"? Or would it be more effective for those two teams to work together on one project?
Part of the problem about KDE vs GTK is that it comes down to C++ vs C, which is a pretty big dividing line in the Linux programmer community. (You can have API wrappers, but the "native" language of the desktop environment influences the core abstractions, and so it'll probably seem more "natural" to use the same language the DE was written in.)
Maybe someone will come along and write a new desktop environment in Rust or something that's better than the other two. My fear that then we'll have three desktop environments.
Another option is to accept that these old debates will never be resolved and to start over with something new that isn't Linux. I've had a feeling like that's the best option that's been growing for the last few years. It's an enormous amount of work, though.
Of course it would be more effective, but it's hard to coordinate individual ambitions and goals within one project, so much so that it will cause splits or over-engineering of features just to please developers.
I don't know if GNU for example can get around that by having Stallman there to give the final say, or the Linux kernel with Torvalds there. Perhaps a similar approach would be good for DEs.
Maybe someone will come along and write a new desktop environment in Rust or something that's better than the other two. My fear that then we'll have three desktop environments.
Another option is to accept that these old debates will never be resolved and to start over with something new that isn't Linux. I've had a feeling like that's the best option that's been growing for the last few years. It's an enormous amount of work, though.