OptiFine's developer is sort of known for not wanting to open up his project and accept outside help. A lot of people are turned off by that but personally I think it is whatever.
The biggest thing is that it is a mess of horribly unoptimized code, hobbled together over more than a decade. In some newer versions, OptiFine can actually perform worse than vanilla. Their rendering changes also cause many issues with other mods, and since it is closed source, developers can't fix them.